System for configuring an optoelectronic sensor

ABSTRACT

Disclosed are methods and apparatus for automatic optoelectronic detection and inspection of objects, based on capturing digital images of a two-dimensional field of view in which an object to be detected or inspected may be located, analyzing the images, and making and reporting decisions on the status of the object. Decisions are based on evidence obtained from a plurality of images for which the object is located in the field of view, generally corresponding to a plurality of viewing perspectives. Evidence that an object is located in the field of view is used for detection, and evidence that the object satisfies appropriate inspection criteria is used for inspection. Methods and apparatus are disclosed for capturing and analyzing images at high speed so that multiple viewing perspectives can be obtained for objects in continuous motion.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.10/865,155 entitled, “Method and Apparatus for Visual Detection andInspection of Objects, filed on Jun. 9, 2004.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to automated detection and inspection of objectsbeing manufactured on a production line, and more particularly to therelated fields of industrial machine vision and automated imageanalysis.

2. Description of the Related Art

Industrial manufacturing relies on automatic inspection of objects beingmanufactured. One form of automatic inspection that has been in commonuse for decades is based on optoelectronic technologies that useelectromagnetic energy, usually infrared or visible light, photoelectricsensors, and some form of electronic decision making.

One well-known form of optoelectronic automatic inspection uses anarrangement of photodetectors. A typical photodetector has a lightsource and a single photoelectric sensor that responds to the intensityof light that is reflected by a point on the surface of an object, ortransmitted along a path that an object may cross. A user-adjustablesensitivity threshold establishes a light intensity above which (orbelow which) an output signal of the photodetector will be energized.

One photodetector, often called a gate, is used to detect the presenceof an object to be inspected. Other photodetectors are arranged relativeto the gate to sense the light reflected by appropriate points on theobject. By suitable adjustment of the sensitivity thresholds, theseother photodetectors can detect whether certain features of the object,such as a label or hole, are present or absent. A decision as to thestatus of the object (for example, pass or fail) is made using theoutput signals of these other photodetectors at the time when an objectis detected by the gate. This decision is typically made by aprogrammable logic controller (PLC), or other suitable electronicequipment.

Automatic inspection using photodetectors has various advantages.Photodetectors are inexpensive, simple to set up, and operate at veryhigh speed (outputs respond within a few hundred microseconds of theobject being detected, although a PLC will take longer to make adecision).

Automatic inspection using photodetectors has various disadvantages,however, including:

-   -   Simple sensing of light intensity reflected from a point on the        object is often insufficient for inspection. Instead it may be        necessary to analyze a pattern of brightness reflected from an        extended area. For example, to detect an edge it may be        necessary to analyze a pattern of brightness to see if it        corresponds to a transition from a lighter to a darker region;    -   It may be hard to arrange the photodetectors when many points on        an object need to be inspected. Each such inspection point        requires the use of a separate photodetector that needs to be        physically mounted in such a way as to not interfere with the        placement of the other photodetectors. Interference may be due        to space limitations, crosstalk from the light sources, or other        factors;    -   Manufacturing lines are usually capable of producing a mix of        products, each with unique inspection requirements. An        arrangement of photodetectors is very inflexible, so that a line        changeover from one product to another would require the        photodetectors to be physically moved and readjusted. The cost        of performing a line changeover, and the risk of human error        involved, often offset the low cost and simplicity of the        photodetectors; and    -   Using an arrangement of photodetectors requires that objects be        presented at known, predetermined locations so that the        appropriate points on the object are sensed. This requirement        may add additional cost and complexity that can offset the low        cost and simplicity of the photodetectors.

Another well-known form of optoelectronic automatic inspection uses adevice that can capture a digital image of a two-dimensional field ofview in which an object to be inspected is located, and then analyze theimage and make decisions. Such a device is usually called a machinevision system, or simply a vision system. The image is captured byexposing a two-dimensional array of photosensitive elements for a briefperiod, called the integration or shutter time, to light that has beenfocused on the array by a lens. The array is called an imager and theindividual elements are called pixels. Each pixel measures the intensityof light falling on it during the shutter time. The measured intensityvalues are then converted to digital numbers and stored in the memory ofthe vision system to form the image, which is analyzed by a digitalprocessing element such as a computer, using methods well-known in theart to determine the status of the object being inspected.

In some cases the objects are brought to rest in the field of view, andin other cases the objects are in continuous motion through the field ofview. An event external to the vision system, such as a signal from aphotodetector, or a message from a PLC, computer, or other piece ofautomation equipment, is used to inform the vision system that an objectis located in the field of view, and therefore an image should becaptured and analyzed. Such an event is called a trigger.

Machine vision systems avoid the disadvantages associated with using anarrangement of photodetectors. They can analyze patterns of brightnessreflected from extended areas, easily handle many distinct features onthe object, accommodate line changeovers through software systems and/orprocesses, and handle uncertain and variable object locations.

Machine vision systems have disadvantages compared to an arrangement ofphotodetectors, including:

-   -   They are relatively expensive, often costing ten times more than        an arrangement of photodetectors;    -   They can be difficult to set up, often requiring people with        specialized engineering training; and    -   They operate much more slowly than an arrangement of        photodetectors, typically requiring tens or hundreds of        milliseconds to make a decision. Furthermore, the decision time        tends to vary significantly and unpredictably from object to        object.

Machine vision systems have limitations that arise because they makedecisions based on a single image of each object, located in a singleposition in the field of view (each object may be located in a differentand unpredictable position, but for each object there is only one suchposition on which a decision is based). This single position providesinformation from a single viewing perspective, and a single orientationrelative to the illumination. The use of only a single perspective oftenleads to incorrect decisions. It has long been observed, for example,that a change in perspective of as little as a single pixel can in somecases change an incorrect decision to a correct one. By contrast, ahuman inspecting an object usually moves it around relative to his eyesand the lights to make a more reliable decision.

Some prior art vision systems capture multiple images of an object atrest in the field of view, and then average those images to produce asingle image for analysis. The averaging reduces measurement noise andthereby improves the decision making, but there is still only oneperspective and illumination orientation, considerable additional timeis needed, and the object must be brought to rest.

Some prior art vision systems that are designed to read alphanumericcodes, bar codes, or 2D matrix codes will capture multiple images andvary the illumination direction until either a correct read is obtained,or all variations have been tried. This method works because such codescontain sufficient redundant information that the vision system can besure when a read is correct, and because the object can be heldstationary in the field of view for enough time to try all of thevariations. The method is generally not suitable for object inspection,and is not suitable when objects are in continuous motion. Furthermore,the method still provides only one viewing perspective, and the decisionis based on only a single image, because information from the imagesthat did not result in a correct read is discarded.

Some prior art vision systems are used to guide robots in pick-and-placeapplications where objects are in continuous motion through the field ofview. Some such systems are designed so that the objects move at a speedin which the vision system has the opportunity to see each object atleast twice. The objective of this design, however, is not to obtain thebenefit of multiple perspectives, but rather to insure that objects arenot missed entirely if conditions arise that temporarily slow down thevision system, such as a higher than average number of objects in thefield of view. These systems do not make use of the additionalinformation potentially provided by the multiple perspectives.

Machine vision systems have additional limitations arising from theiruse of a trigger signal. The need for a trigger signal makes the setupmore complex—a photodetector must be mounted and adjusted, or softwaremust be written for a PLC or computer to provide an appropriate message.When a photodetector is used, which is almost always the case when theobjects are in continuous motion, a production line changeover mayrequire it to be physically moved, which can offset some of theadvantages of a vision system. Furthermore, photodetectors can onlyrespond to a change in light intensity reflected from an object ortransmitted along a path. In some cases, such a condition may not besufficient to reliably detect when an object has entered the field ofview.

Some prior art vision systems that are designed to read alphanumericcodes, bar codes, or two dimensional (2D) matrix codes can operatewithout a trigger by continuously capturing images and attempting toread a code. For the same reasons described above, such methods aregenerally not suitable for object inspection, and are not suitable whenobjects are in continuous motion.

Some prior art vision systems used with objects in continuous motion canoperate without a trigger using a method often called self-triggering.These systems typically operate by monitoring one or more portions ofcaptured images for a change in brightness or color that indicates thepresence of an object. Self-triggering is rarely used in practice due toseveral limitations:

-   -   The vision systems respond too slowly for self-triggering to        work at common production speeds;    -   The methods provided to detect when an object is present are not        sufficient in many cases; and    -   The vision systems do not provide useful output signals that are        synchronized to a specific, repeatable position of the object        along the production line, signals that are typically provided        by the photodetector that acts as a trigger and needed by a PLC        or handling mechanism to take action based on the vision        system's decision.

Many of the limitations of machine vision systems arise in part becausethey operate too slowly to capture and analyze multiple perspectives ofobjects in motion, and too slowly to react to events happening in thefield of view. Since most vision systems can capture a new imagesimultaneously with analysis of the current image, the maximum rate atwhich a vision system can operate is determined by the larger of thecapture time and the analysis time. Overall, one of the most significantfactors in determining this rate is the number of pixels comprising theimager.

The time needed to capture an image is determined primarily by thenumber of pixels in the imager, for two basic reasons. First, theshutter time is determined by the amount of light available and thesensitivity of each pixel. Since having more pixels generally meansmaking them smaller and therefore less sensitive, it is generally thecase that increasing the number of pixels increases the shutter time.Second, the conversion and storage time is proportional to the number ofpixels. Thus the more pixels one has, the longer the capture time.

For at least the last 25 years, prior art vision systems generally haveused about 300,000 pixels; more recently some systems have becomeavailable that use over 1,000,000, and over the years a small number ofsystems have used as few as 75,000. Just as with digital cameras, therecent trend is to more pixels for improved image resolution. Over thesame period of time, during which computer speeds have improved amillion-fold and imagers have changed from vacuum tubes to solid state,machine vision image capture times generally have improved from about1/30 second to about 1/60 second, only a factor of two. Faster computershave allowed more sophisticated analysis, but the maximum rate at whicha vision system can operate has hardly changed.

Recently, CMOS imagers have appeared that allow one to capture a smallportion of the photosensitive elements, reducing the conversion andstorage time. Theoretically such imagers can support very short capturetimes, but in practice, since the light sensitivity of the pixels is nobetter than when the full array is used, it is difficult and/orexpensive to achieve the very short shutter times that would be neededto make such imagers useful at high speed.

Due in part to the image capture time bottleneck, image analysis methodssuited to operating rates significantly higher than 60 images per secondhave not been developed. Similarly, use of multiple perspectives,operation without triggers, production of appropriately synchronizedoutput signals, and a variety of other useful functions have not beenadequately considered in the prior art.

Recently, experimental devices called focal plane array processors havebeen developed in research laboratories. These devices integrate analogsignal processing elements and photosensitive elements on one substrate,and can operate at rates in excess of 10,000 images per second. Theanalog signal processing elements are severely limited in capabilitycompared to digital image analysis, however, and it is not yet clearwhether such devices can be applied to automated industrial inspection.

Considering the disadvantages of an arrangement of photodetectors, andthe disadvantages and limitations of current machine vision systems,there is a compelling need for systems and methods that make use oftwo-dimensional imagers and digital image analysis for improveddetection and inspection of objects in industrial manufacturing.

SUMMARY OF THE INVENTION

The present invention provides systems and methods for automaticoptoelectronic detection and inspection of objects, based on capturingdigital images of a two-dimensional field of view in which an object tobe detected or inspected may be located, and then analyzing the imagesand making decisions. These systems and methods analyze patterns ofbrightness reflected from extended areas, handle many distinct featureson the object, accommodate line changeovers through software means, andhandle uncertain and variable object locations. They are less expensiveand easier to set up than prior art machine vision systems, and operateat much higher speeds. These systems and methods furthermore make use ofmultiple perspectives of moving objects, operate without triggers,provide appropriately synchronized output signals, and provide othersignificant and useful capabilities will become apparent to thoseskilled in the art.

While the present invention is directed primarily at applications wherethe objects are in continuous motion, and provides specific andsignificant advantages in those cases, it may also be usedadvantageously over prior art systems in applications where objects arebrought to rest.

One aspect of the invention is an apparatus, called a vision detector,that can capture and analyze a sequence of images at higher speeds thanprior art vision systems. An image in such a sequence that is capturedand analyzed is called a frame. The rate at which frames are capturedand analyzed, called the frame rate, is sufficiently high that a movingobject is seen in multiple consecutive frames as it passes through thefield of view (FOV). Since the objects moves somewhat between successiveframes, it is located in multiple positions in the FOV, and therefore itis seen from multiple viewing perspectives and positions relative to theillumination.

Another aspect of the invention is a method, called dynamic imageanalysis, for inspecting objects by capturing and analyzing multipleframes for which the object is located in the field of view, and basinga result on a combination of evidence obtained from each of thoseframes. The method provides significant advantages over prior artmachine vision systems that make decisions based on a single frame.

Yet another aspect of the invention is a method, called visual eventdetection, for detecting events that may occur in the field of view. Anevent can be an object passing through the field of view, and by usingvisual event detection the object can be detected without the need for atrigger signal.

Additional aspects of the invention will become apparent by a study ofthe figures and detailed descriptions given herein.

One advantage of the methods and apparatus of the present invention formoving objects is that by considering the evidence obtained frommultiple viewing perspectives and positions relative to theillumination, a vision detector is able to make a more reliable decisionthan a prior art vision system, just as a human inspecting an object maymove it around relative to his eyes and the lights to make a morereliable decision.

Another advantage is that objects can be detected reliably without atrigger signal, such as a photodetector. This reduces cost andsimplifies installation, and allows a production line to be switched toa different product by making a software change in the vision detectorwithout having to manually reposition a photodetector.

Another advantage is that a vision detector can track the position of anobject as it moves through the field of view, and determine its speedand the time at which it crosses some fixed reference point. Outputsignals can then be synchronized to this fixed reference point, andother useful information about the object can be obtained as taughtherein.

In order to obtain images from multiple perspectives, it is desirablethat an object to be detected or inspected moves no more than a smallfraction of the field of view between successive frames, often no morethan a few pixels. As taught herein, it is generally desirable that theobject motion be no more than about one-quarter of the FOV per frame,and in typical embodiments no more than 5% or less of the FOV. It isdesirable that this be achieved not by slowing down a manufacturingprocess but by providing a sufficiently high frame rate. In an examplesystem the frame rate is at least 200 frames/second, and in anotherexample the frame rate is at least 40 times the average rate at whichobjects are presented to the vision detector.

An exemplary system is taught that can capture and analyze up to 500frames/second. This system makes use of an ultra-sensitive imager thathas far fewer pixels than prior art vision systems. The high sensitivityallows very short shutter times using very inexpensive LED illumination,which in combination with the relatively small number of pixels allowsvery short image capture times. The imager is interfaced to a digitalsignal processor (DSP) that can receive and store pixel datasimultaneously with analysis operations. Using methods taught herein andimplemented by means of suitable software for the DSP, the time toanalyze each frame generally can be kept to within the time needed tocapture the next frame. The capture and analysis methods and apparatuscombine to provide the desired high frame rate. By carefully matchingthe capabilities of the imager, DSP, and illumination with theobjectives of the invention, the exemplary system can be significantlyless expensive than prior art machine vision systems.

The method of visual event detection involves capturing a sequence offrames and analyzing each frame to determine evidence that an event isoccurring or has occurred. When visual event detection used to detectobjects without the need for a trigger signal, the analysis woulddetermine evidence that an object is located in the field of view.

In an exemplary method the evidence is in the form of a value, called anobject detection weight, that indicates a level of confidence that anobject is located in the field of view. The value may be a simple yes/nochoice that indicates high or low confidence, a number that indicates arange of levels of confidence, or any item of information that conveysevidence. One example of such a number is a so-called fuzzy logic value,further described herein. Note that no machine can make a perfectdecision from an image, and so will instead make judgments based onimperfect evidence.

When performing object detection, a test is made for each frame todecide whether the evidence is sufficient that an object is located inthe field of view. If a simple yes/no value is used, the evidence may beconsidered sufficient if the value is “yes”. If a number is used,sufficiency may be determined by comparing the number to a threshold.Frames where the evidence is sufficient are called active frames. Notethat what constitutes sufficient evidence is ultimately defined by ahuman user who configures the vision detector based on an understandingof the specific application at hand. The vision detector automaticallyapplies that definition in making its decisions.

When performing object detection, each object passing through the fieldof view will produce multiple active frames due to the high frame rateof the vision detector. These frames may not be strictly consecutive,however, because as the object passes through the field of view theremay be some viewing perspectives, or other conditions, for which theevidence that the object is located in the field of view is notsufficient. Therefore it is desirable that detection of an object beginswhen a active frame is found, but does not end until a number ofconsecutive inactive frames are found. This number can be chosen asappropriate by a user.

Once a set of active frames has been found that may correspond to anobject passing through the field of view, it is desirable to perform afurther analysis to determine whether an object has indeed beendetected. This further analysis may consider some statistics of theactive frames, including the number of active frames, the sum of theobject detection weights, the average object detection weight, and thelike.

The above examples of visual event detection are intended to beillustrative and not comprehensive. Clearly there are many ways toaccomplish the objectives of visual event detection within the spirit ofthe invention that will occur to one of ordinary skill.

The method of dynamic image analysis involves capturing and analyzingmultiple frames to inspect an object, where “inspect” means to determinesome information about the status of the object. In one example of thismethod, the status of an object includes whether or not the objectsatisfies inspection criteria chosen as appropriate by a user.

In some aspects of the invention dynamic image analysis is combined withvisual event detection, so that the active frames chosen by the visualevent detection method are the ones used by the dynamic image analysismethod to inspect the object. In other aspects of the invention, theframes to be used by dynamic image analysis can be captured in responseto a trigger signal.

Each such frame is analyzed to determine evidence that the objectsatisfies the inspection criteria. In one exemplary method, the evidenceis in the form of a value, called an object pass score, that indicates alevel of confidence that the object satisfies the inspection criteria.As with object detection weights, the value may be a simple yes/nochoice that indicates high or low confidence, a number, such as a fuzzylogic value, that indicates a range of levels of confidence, or any itemof information that conveys evidence.

The status of the object may be determined from statistics of the objectpass scores, such as an average or percentile of the object pass scores.The status may also be determined by weighted statistics, such as aweighted average or weighted percentile, using the object detectionweights. Weighted statistics effectively weight evidence more heavilyfrom frames wherein the confidence is higher that the object is actuallylocated in the field of view for that frame.

Evidence for object detection and inspection is obtained by examining aframe for information about one or more visible features of the object.A visible feature is a portion of the object wherein the amount,pattern, or other characteristic of emitted light conveys informationabout the presence, identity, or status of the object. Light can beemitted by any process or combination of processes, including but notlimited to reflection, transmission, or refraction of a source externalor internal to the object, or directly from a source internal to theobject.

One aspect of the invention is a method for obtaining evidence,including object detection weights and object pass scores, by imageanalysis operations on one or more regions of interest in each frame forwhich the evidence is needed. In example of this method, the imageanalysis operation computes a measurement based on the pixel values inthe region of interest, where the measurement is responsive to someappropriate characteristic of a visible feature of the object. Themeasurement is converted to a logic value by a threshold operation, andthe logic values obtained from the regions of interest are combined toproduce the evidence for the frame. The logic values can be binary orfuzzy logic values, with the thresholds and logical combination beingbinary or fuzzy as appropriate.

For visual event detection, evidence that an object is located in thefield of view is effectively defined by the regions of interest,measurements, thresholds, logical combinations, and other parametersfurther described herein, which are collectively called theconfiguration of the vision detector and are chosen by a user asappropriate for a given application of the invention. Similarly, theconfiguration of the vision detector defines what constitutes sufficientevidence.

For dynamic image analysis, evidence that an object satisfies theinspection criteria is also effectively defined by the configuration ofthe vision detector.

One aspect of the invention includes determining a result comprisinginformation about detection or inspection of an object. The result maybe reported to automation equipment for various purposes, includingequipment, such as a reject mechanism, that may take some action basedon the report. In one example the result is an output pulse that isproduced whenever an object is detected. In another example, the resultis an output pulse that is produced only for objects that satisfy theinspection criteria. In yet another example, useful for controlling areject actuator, the result is an output pulse that is produced only forobjects that do not satisfy the inspection criteria.

Another aspect of the invention is a method for producing output signalsthat are synchronized to a time, shaft encoder count, or other eventmarker that indicates when an object has crossed a fixed reference pointon a production line. A synchronized signal provides information aboutthe location of the object in the manufacturing process, which can beused to advantage by automation equipment, such as a downstream rejectactuator.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be more fully understood from the following detaileddescription, in conjunction with the accompanying figures, wherein:

FIG. 1 shows a prior art machine vision system used to inspect objectson a production line;

FIG. 2 shows a timeline that illustrates a typical operating cycle of aprior art machine vision system;

FIG. 3 shows prior art inspection of objects on a production line usingphotodetectors;

FIG. 4 shows an illustrative embodiment of a vision detector accordingto the present invention, inspecting objects on a production line;

FIG. 5 shows a timeline that illustrates a typical operating cycle for avision detector using visual event detection;

FIG. 6 shows a timeline that illustrates a typical operating cycle for avision detector using a trigger signal;

FIG. 7 shows a timeline that illustrates a typical operating cycle for avision detector performing continuous analysis of a manufacturingprocess;

FIG. 8 shows a high-level block diagram for a vision detector in aproduction environment;

FIG. 9 shows a block diagram of an illustrative embodiment of a visiondetector;

FIG. 10 shows illumination arrangements suitable for a vision detector;

FIG. 11 shows fuzzy logic elements used in an illustrative embodiment toweigh evidence and make judgments, including judging whether an objectis present and whether it passes inspection;

FIG. 12 illustrates how evidence is weighed for dynamic image analysisin an illustrative embodiment;

FIG. 13 illustrates how evidence is weighed for dynamic image analysisin another illustrative embodiment;

FIG. 14 shows the organization of a set of software elements (e.g.,program instructions of a computer readable medium) used by anillustrative embodiment to analyze frames, make judgments, sense inputs,and control output signals;

FIG. 15 shows a portion of an exemplary configuration of a visiondetector that may be used to inspect an exemplary object;

FIG. 16 shows another portion of the configuration corresponding to theexemplary setup of FIG. 15;

FIG. 17 illustrates a method for analyzing regions of interest tomeasure brightness and contrast of a visible feature;

FIG. 18 illustrates a method for analyzing regions of interest to detectstep edges;

FIG. 19 further illustrates a method for analyzing regions of interestto detect step edges;

FIG. 20 illustrates a method for analyzing regions of interest to detectridge edges;

FIG. 21 further illustrates a method for analyzing regions of interestto detect ridge edges, and illustrates a method for detecting eitherstep or ridge edges;

FIG. 22 shows graphical controls that can be displayed on anhuman-machine interface (HMI) for a user to view and manipulate in orderto set parameters for detecting edges;

FIG. 23 illustrates a method for analyzing regions of interest to detectspots;

FIG. 24 further illustrates a method for analyzing regions of interestto detect spots;

FIG. 25 illustrates a method for analyzing regions of interest to trackthe location of objects in the field of view, and using an HMI toconfigure the analysis;

FIG. 26 further illustrates a method for analyzing regions of interestto track the location of objects in the field of view, and using an HMIto configure the analysis;

FIG. 27 further illustrates a method for analyzing regions of interestto track the location of objects in the field of view, and using an HMIto configure the analysis;

FIG. 28 illustrates a method for analyzing regions of interest to trackthe location of objects in the field of view, and using an HMI toconfigure the analysis, in certain cases where placement of the regionsof interest must be fairly precise along an object boundary;

FIG. 29 illustrates a method for analyzing regions of interest to trackthe location of objects in the field of view, and using an HMI toconfigure the analysis, in cases where objects may rotation and changesize;

FIG. 30 shows an alternative representation for a portion of theconfiguration of a vision detector, based on a ladder diagram, a widelyused industrial programming language;

FIG. 31 shows a timing diagram that will be used to explain how visiondetector output signals are synchronized;

FIG. 32 shows an example of measuring the time at which an objectcrosses a fixed reference point, and also measuring object speed, pixelsize calibration, and object distance and attitude in space;

FIG. 33 shows an output buffer that is used to generate synchronizedpulses for downstream control of actuators;

FIG. 34 shows a portion of the HMI for user configuration of objectdetection parameters;

FIG. 35 shows a portion of the HMI for user configuration of objectinspection parameters;

FIG. 36 shows a portion of the HMI for user configuration of outputsignals;

FIG. 37 illustrates one way to configure the invention to perform visualevent detection when connected to a PLC;

FIG. 38 illustrates one way to configure the invention to perform visualevent detection for direct control of a reject actuator;

FIG. 39 illustrates one way to configure the invention to use a triggersignal when connected to a PLC;

FIG. 40 illustrates one way to configure the invention to use a triggersignal for direct control of a reject actuator;

FIG. 41 illustrates one way to configure the invention to performcontinuous analysis for detection of flaws on a continuous web;

FIG. 42 illustrates one way to configure the invention for detection offlaws on a continuous web that provides for signal filtering;

FIG. 43 illustrates one way to configure the invention for detection offlaws on a continuous web that provides for signal synchronization;

FIG. 44 illustrates one way to configure the invention to perform visualevent detection without tracking the location of the object in the fieldof view;

FIG. 45 illustrates rules that are used by an illustrative embodiment ofa vision detector for learning appropriate parameter settings based onexamples shown by a user;

FIG. 46 further illustrates rules that are used by an illustrativeembodiment of a vision detector for learning appropriate parametersettings based on examples shown by a user;

FIG. 47 illustrates the use of a phase-locked loop (PLL) to measure theobject presentation rate, and to detect missing and extra objects, forproduction lines that present objects at an approximately constant rate;and

FIG. 48 illustrates the operation of a novel software PLL used in anillustrative embodiment.

DETAILED DESCRIPTION OF THE INVENTION

Discussion of Prior Art

FIG. 1 shows a prior art machine vision system used to inspect objectson a production line. Objects 110, 112, 114, 116, and 118 move left toright on a conveyer 100. Each object is expected to contain certainfeatures, for example a label 120 and a hole 124. Objects incorrectlymanufactured may be missing one or more features, or may have unintendedfeatures; for example, object 116 is missing the hole. On manyproduction lines motion of the conveyer is tracked by a shaft encoder180, which sends a signal 168 to a programmable logic controller (PLC)140.

The objects move past a photodetector 130, which emits a beam of light135 for detecting the presence of an object. Trigger signals 162 and 166are sent from the photodetector to the PLC 140, and a machine visionsystem 150. On the leading edge of the trigger signal 166 the visionsystem 150 captures an image of the object, inspects the image todetermine if the expected features are present, and reports theinspection results to the PLC via signal 160.

On the leading edge of the trigger signal 162 the PLC records the timeand/or encoder count. At some later time the PLC receives the results ofthe inspection from the vision system, and may do various things withthose results as appropriate. For example the PLC may control a rejectactuator 170 via signal 164 to remove a defective object 116 from theconveyer. Since the reject actuator is generally downstream from theinspection point defined by the photodetector beam 135, the PLC mustdelay the signal 164 to the reject actuator until the defective part isin position in front of the reject actuator. Since the time it takes thevision system to complete the inspection is usually somewhat variable,this delay must be relative to the trigger signal 162, i.e. relative tothe time and/or count recorded by the PLC. A time delay is appropriatewhen the conveyer is moving at constant speed; in other cases, anencoder is preferred.

FIG. 1 does not show illumination, which would be provided asappropriate according to various methods well-known in the art.

In the example of FIG. 1, the objects are in continuous motion. Thereare also many applications for which the production equipment bringsobjects to rest in front of the vision system.

FIG. 2 shows a timeline that illustrates a typical operating cycle of aprior art machine vision system. Shown are the operating steps for twoexemplary objects 200 and 210. The operating cycle contains four steps:trigger 220, image capture 230, analyze 240, and report 250. During thetime between cycles 260, the vision system is idle. The timeline is notdrawn to scale, and the amount of time taken by the indicated steps willvary significantly among applications.

The trigger 220 is some event external to the vision system, such as asignal from a photodetector 130, or a message from a PLC, computer, orother piece of automation equipment.

The image capture step 230 starts by exposing a two-dimensional array ofphotosensitive elements, called pixels, for a brief period, called theintegration or shutter time, to an image that has been focused on thearray by a lens. Each pixel measures the intensity of light falling onit during the shutter time. The measured intensity values are thenconverted to digital numbers and stored in the memory of the visionsystem.

During the analyze step 240 the vision system operates on the storedpixel values using methods well-known in the art to determine the statusof the object being inspected. During the report step 250, the visionsystem communicates information about the status of the object toappropriate automation equipment, such as a PLC.

FIG. 3 shows prior art inspection of objects on a production line usingphotodetectors. Conveyer 100, objects 110, 112, 114, 116, 118, label120, hole 124, encoder 180, reject actuator 170, and signals 164 and 168are as described for FIG. 1. A first photodetector 320 with beam 325 isused to detect the presence of an object. A second photodetector 300with beam 305 is positioned relative to the first photodetector 320 soas to be able to detect the presence of label 120. A third photodetector310 with beam 315 is positioned relative to the first photodetector 320so as to be able to detect the presence of hole 124.

A PLC 340 samples signals 330 and 333 from photodetectors 300 and 310 onthe leading edge of signal 336 from photodetector 330 to determine thepresence of features 120 and 124. If one or both features are missing,signal 164 is sent to reject actuator 170, suitably delayed based onencoder 180, to remove a defective object from the conveyer.

Basic Operation of Present Invention

FIG. 4 shows an illustrative embodiment of a vision detector accordingto the present invention, inspecting objects on a production line. Aconveyer 100 transports objects to cause relative movement between theobjects and the field of view of vision detector 400. Objects 110, 112,114, 116, 118, label 120, hole 124, encoder 180, and reject actuator 170are as described for FIG. 1. A vision detector 400 detects the presenceof an object by visual appearance and inspects it based on appropriateinspection criteria. If an object is defective, the vision detectorsends signal 420 to reject actuator 170 to remove the object from theconveyer stream. The encoder 180 sends signal 410 to the visiondetector, which uses it to insure proper delay of signal 420 from theencoder count where the object crosses some fixed, imaginary referencepoint 430, called the mark point. If an encoder is not used, the delaycan be based on time instead.

In an alternate embodiment, the vision detector sends signals to a PLCfor various purposes, which may include controlling a reject actuator.

In another embodiment, suitable in extremely high speed applications orwhere the vision detector cannot reliably detect the presence of anobject, a photodetector is used to detect the presence of an object andsends a signal to the vision detector for that purpose.

In yet another embodiment there are no discrete objects, but rathermaterial flows past the vision detector continuously, for example a web.In this case the material is inspected continuously, and signals aresend by the vision detector to automation equipment, such as a PLC, asappropriate.

When a vision detector detects the presence of discrete objects byvisual appearance, it is said to be operating in visual event detectionmode. When a vision detector detects the presence of discrete objectsusing an external signal such as from a photodetector, it is said to beoperating in external trigger mode. When a vision detector continuouslyinspects material, it is said to be operating in continuous analysismode.

FIG. 5 shows a timeline that illustrates a typical operating cycle for avision detector in visual event detection mode. Boxes labeled “c”, suchas box 520, represent image capture. Boxes labeled “a”, such as box 530,represent image analysis. It is desirable that capture “c” of the nextimage be overlapped with analysis “a” of the current image, so that (forexample) analysis step 530 analyzes the image captured in capture step520. In this timeline, analysis is shown as taking less time thancapture, but in general analysis will be shorter or longer than capturedepending on the application details.

If capture and analysis are overlapped, the rate at which a visiondetector can capture and analyze images is determined by the longer ofthe capture time and the analysis time. This is the “frame rate”.

The present invention allows objects to be detected reliably without atrigger signal, such as that provided by a photodetector 130. Referringto FIG. 4, there is no trigger signal to indicate the presence of anobject, and in FIG. 5 there is no corresponding trigger step such asstep 220 in FIG. 2.

Referring again to FIG. 5, a portion 500 of the timeline corresponds tothe inspection of a first object, and contains the capture and analysisof seven frames. A second portion 510 corresponds to the inspection of asecond object, and contains five frames.

Each analysis step first considers the evidence that an object ispresent. Frames where the evidence is sufficient are called active.Analysis steps for active frames are shown with a thick border, forexample analysis step 540. In an illustrative embodiment, inspection ofan object begins when an active frame is found, and ends when somenumber of consecutive inactive frames are found. In the example of FIG.5, inspection of the first object begins with the first active framecorresponding to analysis step 540, and ends with two consecutiveinactive frames, corresponding to analysis steps 546 and 548. Note thatfor the first object, a single inactive frame corresponding to analysisstep 542 is not sufficient to terminate the inspection.

At the time that inspection of an object is complete, for example at theend of analysis step 548, decisions are made on the status of the objectbased on the evidence obtained from the active frames. In anillustrative embodiment, if an insufficient number of active frames werefound then there is considered to be insufficient evidence that anobject was actually present, and so operation continues as if no activeframes were found. Otherwise an object is judged to have been detected,and evidence from the active frames is judged in order to determine itsstatus, for example pass or fail. A variety of methods may be used todetect objects and determine status within the scope of the invention;some are described below and many others will occur to those skilled inthe art.

Once an object has been detected and a judgment made, a report may bemade to appropriate automation equipment, such as a PLC, using signalswell-known in the art. In such a case a report step similar to step 250in FIG. 2 would appear in the timeline. The example of FIG. 5corresponds instead to a setup such as shown in FIG. 4, where the visiondetector is used to control a downstream reject actuator 170 via signal420. By considering the position of the object in the active frames asit passes through the field of view, the vision detector estimates themark time 550 and 552 at which the object crosses the mark point 430.Note that in cases where an encoder 180 is used, the mark time isactually an encoder count; the reader will understand that time andcount can be used interchangeably. A report 560, consisting of a pulseof appropriate duration to the reject actuator 170, is issued after aprecise delay 570 in time or encoder count from the mark time 550.

Note that the report 560 may be delayed well beyond the inspection ofsubsequent objects such as 510. The vision detector uses well-knownfirst-in first-out (FIFO) buffer methods to hold the reports until theappropriate time.

Once inspection of an object is complete, the vision detector may enteran idle step 580. Such a step is optional, but may be desirable forseveral reasons. If the maximum object rate is known, there is no needto be looking for an object until just before a new one is due. An idlestep will eliminate the chance of false object detection at times whenan object couldn't arrive, and will extend the lifetime of theillumination system because the lights can be kept off during the idlestep.

FIG. 6 shows a timeline that illustrates a typical operating cycle for avision detector in external trigger mode. A trigger step 620, similar infunction to prior art trigger step 220, begins inspection of a firstobject 600. A sequence of image capture steps 630, 632, and 634, andcorresponding analysis steps 640, 642, and 644 are used for dynamicimage analysis. As in visual event detection mode, it is desirable thatthe frame rate be sufficiently high that the object moves a smallfraction of the field of view between successive frames, often no morethan a few pixels per frame. After a fixed number of frames, the numberbeing chosen based on application details, the evidence obtained fromanalysis of the frames is used to make a final judgment on the status ofthe object, which in one embodiment is provided to automation equipmentin a report step 650. Following the report step, an idle step 660 isentered until the next trigger step 670 that begins inspection of asecond object 610.

In another embodiment, the report step is delayed in a manner equivalentto that shown in FIG. 5. In this embodiment, the mark time 680 is thetime (or encoder count) corresponding to the trigger step 620.

FIG. 7 shows a timeline that illustrates a typical operating cycle for avision detector in continuous analysis mode. Frames are captured,analyzed, and reported continuously. One such cycle comprises capturestep 700, analysis step 710, and report step 720.

Illustrative Apparatus

FIG. 8 shows a high-level block diagram for a vision detector in aproduction environment. A vision detector 800 is connected toappropriate automation equipment 810, which may include PLCs, rejectactuators, and/or photodetectors, by means of signals 820. The visiondetector may also be connected to a human-machine interface (HMI) 830,such as a PC or hand-held device, by means of signals 840. The HMI isused for setup and monitoring, and may be removed during normalproduction use. The signals can be implemented in any acceptable formatand/or protocol and transmitted in a wired or wireless form.

FIG. 9 shows a block diagram of an illustrative embodiment of a visiondetector. A digital signal processor (DSP) 900 runs software to controlcapture, analysis, reporting, HMI communications, and any otherappropriate functions needed by the vision detector. The DSP 900 isinterfaced to a memory 910, which includes high speed random accessmemory for programs and data and non-volatile memory to hold programsand setup information when power is removed. The DSP is also connectedto an I/O module 920 that provides signals to automation equipment, anHMI interface 930, an illumination module 940, and an imager 960. A lens950 focuses images onto the photosensitive elements of the imager 960.

The DSP 900 can be any device capable of digital computation,information storage, and interface to other digital elements, includingbut not limited to a general-purpose computer, a PLC, or amicroprocessor. It is desirable that the DSP 900 be inexpensive but fastenough to handle a high frame rate. It is further desirable that it becapable of receiving and storing pixel data from the imagersimultaneously with image analysis.

In the illustrative embodiment of FIG. 9, the DSP 900 is an ADSP-BF531manufactured by Analog Devices of Norwood, Mass. The Parallel PeripheralInterface (PPI) 970 of the ADSP-BF531 DSP 900 receives pixel data fromthe imager 960, and sends the data to memory controller 974 via DirectMemory Access (DMA) channel 972 for storage in memory 910. The use ofthe PPI 970 and DMA 972 allows, under appropriate software control,image capture to occur simultaneously with any other analysis performedby the DSP 900. Software instructions to control the PPI 970 and DMA 972can be implemented by one of ordinary skill in the art following theprogramming instructions contained in the ADSP-BF533 Blackfin ProcessorHardware Reference (part number 82-002005-01), and the BlackfinProcessor Instruction Set Reference (part number 82-000410-14), bothincorporated herein by reference. Note that the ADSP-BF531, and thecompatible ADSP-BF532 and ADSP-BF533 devices, have identical programminginstructions and can be used interchangeably in this illustrativeembodiment to obtain an appropriate price/performance tradeoff.

The high frame rate desired by a vision detector suggests the use of animager unlike those that have been used in prior art vision systems. Itis desirable that the imager be unusually light sensitive, so that itcan operate with extremely short shutter times using inexpensiveillumination. It is further desirable that it be able to digitize andtransmit pixel data to the DSP far faster than prior art vision systems.It is moreover desirable that it be inexpensive and have a globalshutter.

These objectives may be met by choosing an imager with much higher lightsensitivity and lower resolution than those used by prior art visionsystems. In the illustrative embodiment of FIG. 9, the imager 960 is anLM9630 manufactured by National Semiconductor of Santa Clara, Calif. TheLM9630 has an array of 128 by 100 pixels, for a total of 12800, about 24times fewer than typical prior art vision systems. The pixels arerelatively large at 20 microns square, providing high light sensitivity.The LM9630 can provide 500 frames per second when set for a 300microsecond shutter time, and is sensitive enough (in most cases) toallow a 300 microsecond shutter using LED illumination. This resolutionwould be considered far too low for a vision system, but is quitesufficient for the feature detection tasks that are the objectives ofthe present invention. Electrical interface and software control of theLM9630 can be implemented by one of ordinary skill in the art followingthe instructions contained in the LM9630 Data Sheet, Rev 1.0, January2004, which is incorporated herein by reference.

It is desirable that the illumination 940 be inexpensive and yet brightenough to allow short shutter times. In an illustrative embodiment, abank of high-intensity red LEDs operating at 630 nanometers is used, forexample the HLMP-ED25 manufactured by Agilent Technologies. In anotherembodiment, high-intensity white LEDs are used to implement desiredillumination.

In the illustrative embodiment of FIG. 9, the I/O module 920 providesoutput signals 922 and 924, and input signal 926. One such output signalcan be used to provide signal 420 (FIG. 4) for control of rejectactuator 170. Input signal 926 can be used to provide an externaltrigger.

As used herein an image capture device provides means to capture andstore a digital image. In the illustrative embodiment of FIG. 9, imagecapture device 980 comprises a DSP 900, imager 960, memory 910, andassociated electrical interfaces and software instructions.

As used herein an analyzer provides means for analysis of digital data,including but not limited to a digital image. In the illustrativeembodiment of FIG. 9, analyzer 982 comprises a DSP 900, a memory 910,and associated electrical interfaces and software instructions.

As used herein an output signaler provides means to produce an outputsignal responsive to an analysis. In the illustrative embodiment of FIG.9, output signaler 984 comprises an I/O module 920 and an output signal922.

It will be understood by one of ordinary skill that there are manyalternate arrangements, devices, and software instructions that could beused within the scope of the present invention to implement an imagecapture device 980, analyzer 982, and output signaler 984.

A variety of engineering tradeoffs can be made to provide efficientoperation of an apparatus according to the present invention for aspecific application. Consider the following definitions:

-   -   b fraction of the FOV occupied by the portion of the object that        contains the visible features to be inspected, determined by        choosing the optical magnification of the lens 950 so as to        achieve good use of the available resolution of imager 960;    -   e fraction of the FOV to be used as a margin of error;    -   n desired minimum number of frames in which each object will        typically be seen;    -   s spacing between objects as a multiple of the FOV, generally        determined by manufacturing conditions;    -   p object presentation rate, generally determined by        manufacturing conditions;    -   m maximum fraction of the FOV that the object will move between        successive frames, chosen based on above values; and    -   r minimum frame rate, chosen based on above values.

From these definitions it can be seen that

$\begin{matrix}{m \leq \frac{1 - b - e}{n}} & (1) \\\text{and} & \; \\{r \geq \frac{sp}{m}} & (2)\end{matrix}$

To achieve good use of the available resolution of the imager, it isdesirable that b is at least 50%. For dynamic image analysis, n shouldbe at least 2. Therefore it is further desirable that the object movesno more than about one-quarter of the field of view between successiveframes.

In an illustrative embodiment, reasonable values might be b=75%, e=5%,and n=4. This implies that m≦5%, i.e. that one would choose a frame rateso that an object would move no more than about 5% of the FOV betweenframes. If manufacturing conditions were such that s=2, then the framerate r would need to be at least approximately 40 times the objectpresentation rate p. To handle an object presentation rate of 5 Hz,which is fairly typical of industrial manufacturing, the desired framerate would be at least around 200 Hz. This rate could be achieved usingan LM9630 with at most a 3.3 millisecond shutter time, as long as theimage analysis is arranged so as to fit within the 5 millisecond frameperiod. Using available technology, it would be feasible to achieve thisrate using an imager containing up to about 40,000 pixels.

With the same illustrative embodiment and a higher object presentationrate of 12.5 Hz, the desired frame rate would be at least approximately500 Hz. An LM9630 could handle this rate by using at most a 300microsecond shutter.

In another illustrative embodiment, one might choose b=75%, e=15%, andn=5, so that m≦2%. With s=2 and p=5 Hz, the desired frame rate wouldagain be at least approximately 500 Hz.

FIG. 10 shows illumination arrangements suitable for a vision detector.In one embodiment, a ring 1000 of 18 LEDs, including exemplary LED 1010,surrounds a lens 1030. The ring is divided into six banks of three LEDs,including example bank 1020, that can be controlled independently. Theindependent control of the banks allows the illumination direction to beadjusted as appropriate for a given application.

In an alternate embodiment, a rectangular array 1050 of 16 LEDs,including exemplary LED 1060, surrounds a lens 1080. The array isdivided into four banks as shown, including example bank 1070.

The capabilities of dynamic image analysis according to the presentinvention can be enhanced in some cases by controlling the LEDs so thatsuccessive frames are captured with varying banks illuminated. Byconsidering the evidence obtained from frames illuminated from varyingdirections, it is possible to reliably detect features that would bedifficult to detect with a fixed illumination direction. Accordingly,the present invention allows analysis utilizing varying directionillumination with moving objects.

Fuzzy Logic Decision Making

FIG. 11 shows fuzzy logic elements used in an illustrative embodiment toweigh evidence and make judgments, including judging whether an objectis present and whether it passes inspection.

A fuzzy logic value is a number between 0 and 1 that represents anestimate of confidence that some specific condition is true. A value of1 signifies high confidence that the condition is true, 0 signifies highconfidence that the condition is false, and intermediate values signifyintermediate levels of confidence.

The more familiar binary logic is a subset of fuzzy logic, where theconfidence values are restricted to just 0 and 1. Therefore, anyembodiment described herein that uses fuzzy logic values can use as analternative binary logic values, with any fuzzy logic method orapparatus using those values replaced with an equivalent binary logicmethod or apparatus.

Just as binary logic values are obtained from raw measurements by usinga threshold, fuzzy logic values are obtained using a fuzzy threshold.Referring to FIG. 11, a graph 1100 illustrates a fuzzy threshold. Thex-axis 1110 represents a raw measurement, and the f-axis 1114 representsthe fuzzy logic value, which is a function whose domain includes allpossible raw measurements and whose range is 0≦f≦1.

In an illustrative embodiment, a fuzzy threshold comprises two numbersshown on the x-axis, low threshold t₀ 1120, and high threshold t₁ 1122,corresponding to points on the function 1124 and 1126. The fuzzythreshold can be defined by the equation

$\begin{matrix}{f = {\min\left( {{\max\left( {\frac{x - t_{0}}{t_{1} - t_{0}},0} \right)},1} \right)}} & (3)\end{matrix}$

Note that this function works just as well when t₁<t₀. Other functionscan also be used for a fuzzy threshold, such as the sigmoid

$\begin{matrix}{f = \frac{1}{1 + {\mathbb{e}}^{{- {({x - t})}}/\sigma}}} & (4)\end{matrix}$

where t and σ are threshold parameters. In embodiments where simplicityis a goal, a conventional binary threshold can be used, resulting inbinary logic values.

Fuzzy decision making is based on fuzzy versions of AND 1140, OR 1150,and NOT 1160. A fuzzy AND of two or more fuzzy logic values is theminimum value, and a fuzzy OR is the maximum value. Fuzzy NOT of f is1−f Fuzzy logic is identical to binary when the fuzzy logic values arerestricted to 0 and 1.

In an illustrative embodiment, whenever a hard true/false decision isneeded, a fuzzy logic value is considered true if it is at least 0.5,false if it is less than 0.5.

It will be clear to one skilled in the art that there is nothingcritical about the values 0 and 1 as used in connection with fuzzy logicherein. Any number could be used to represent high confidence that acondition is true, and any different number could be used to representhigh confidence that the condition is false, with intermediate valuesrepresenting intermediate levels of confidence.

Dynamic Image Analysis

FIG. 12 illustrates how evidence is weighed for dynamic image analysisin an illustrative embodiment. In this embodiment two decisions, calledthe primary decisions, must be made:

1. Is an object, or a set of visible features of an object, located inthe field of view?

2. If so, what is the status of the object?

Information comprising evidence that an object is located in the fieldof view is called an object detection weight. Information comprisingevidence regarding the status of an object is called an object passscore. In various illustrative embodiments, the status of the objectcomprises whether or not the object satisfies inspection criteria chosenas appropriate by a user. In the following, an object that satisfies theinspection criteria is sometimes said to “pass inspection”.

FIG. 12 shows two plots, object detection plot 1200 and object pass plot1202. The horizontal axes of the two plots represent a frame sequencenumber i; each frame is represented by a vertical line, such as exampleline 1204.

In the illustrative embodiment of FIG. 12, object detection weights arefuzzy logic values d_(i), representing evidence that an object islocated in the FOV in frame i, and are computed by the vision detectoron each frame using methods further described below. Object pass scoresare fuzzy logic values p_(i), representing evidence that an objectsatisfies appropriate inspection criteria in frame i, and are computedby the vision detector on selected frames using methods furtherdescribed below. The vertical axis of object detection plot 1200represents d_(i), and the vertical axis of object pass plot 1202represents p_(i).

In the illustrative embodiment of FIG. 12, frames where d_(i)≧0.5 areconsidered active (refer to the above description of FIG. 5 for anexplanation of active frames). For reference, a line 1230 whered_(i)=0.5 is plotted. Object detection weights and pass scores foractive frames are plotted as solid circles, for example points 1210 and1212, and those for inactive frames are plotted as open circles, forexample points 1214 and 1216. In some embodiments, the object passweights are only computed for active frames; while in the embodiment ofFIG. 12, they are computed for all frames, whether or not deemed“active”.

In the example of FIG. 12, all of the active frames correspond to theinspection of a single object; as explained in the above description ofFIG. 5, the isolated inactive frame 1220 does not terminate theinspection.

In one embodiment, an object is judged to have been detected if thenumber of active frames found exceeds some threshold. An anotherembodiment, an object is judged to have been detected if the totalobject detection weight over all active frames exceeds some threshold.These thresholds are set as appropriate for a given application (seeFIG. 34).

In the illustrative embodiment of FIG. 12, an object is judged to passinspection if the weighted average of the object pass scores, eachweighted by the corresponding object detection weight, is at least 0.5.More precisely, the object passes inspection if

$\begin{matrix}{\frac{\sum\limits_{i}{d_{i}p_{i}}}{\sum\limits_{i}d_{i}} \geq 0.5} & (5)\end{matrix}$

where the summation is over all active frames. The effect of thisformula is to average the object pass scores, but to weight each scorebased on the confidence that the object really did appear in thecorresponding frame.

In an alternate embodiment, an object is judged to pass inspection ifthe average of the object pass scores is at least 0.5. This isequivalent to a weighted average wherein all of the weights are equal.

In the example of FIG. 12, the weighted average pass score is around0.86, which is plotted as line 1240. The number of active frames is 11,and the total object detection weight is around 9.5. In this example anobject is detected and passes inspection.

FIG. 13 illustrates how evidence is weighed for dynamic image analysisin another illustrative embodiment. In this example, object featuresbeing inspected are difficult to see and only appear with confidence ina small number of the active frames, primarily frames 1300 and 1310,when the viewing and illumination perspective is just right as theobject moves through the field of view. As long as the evidence issufficient in these few frames that the features are present, the objectshould pass inspection. In this scenario, there is no way to know inadvance which of the active frames will contain this evidence. Thus,weighted average pass score is not appropriate in this case. Analternative is to pass the object if the pass score exceeds somethreshold in any of the active frames, but this alternative may passobjects on the basis of too little evidence. In the illustrativeembodiment of FIG. 13, a weighted percentile method is used.

The weighted percentile method is based on the fraction Q(p) of totalweight where the pass score is at least p:

$\begin{matrix}{{Q(p)} = \frac{\sum\limits_{k❘{p_{k} \geq p}}d_{k}}{\sum\limits_{i}d_{i}}} & (6)\end{matrix}$

The object is judged to pass if Q(p) is at least some threshold t. Inthe illustrative embodiment of FIG. 13, p=0.5, which is plotted as line1320. A reasonable threshold t for this case would be 10%.

Useful behavior is obtained using different values of t. For example, ift=50%, the object is judged to pass inspection if the weighted medianscore is at least p. Weighted median is similar to weighted average, butwith properties more appropriate in some cases. For higher values, forexample t=90%, the object will be judged to pass inspection only if theoverwhelming majority of the weight corresponds to active frames wherethe pass score is at least p. For t=100%, the object will be judged topass inspection only if all of the active frames have a pass score thatis at least p. The object may also be judged to pass inspection if Q(p)is greater than 0, which means that any active frame has frame a passscore that is at least p.

In another useful variation, the object is judged to pass inspectionbased on the total weight where the pass score is at least p, instead ofthe fraction of total weight.

In an alternate embodiment, a percentile method is used based on a countof the frames where the pass score is at least p. This is equivalent toa weighted percentile method wherein all of the weights are equal.

The above descriptions of methods for weighing evidence to determinewhether an object has been detected, and whether it passes inspection,are intended as examples of useful embodiments, but do not limit themethods that can be used within the scope of the invention. For example,the exemplary constants 0.5 used above may be replaced with any suitablevalue. Many additional methods for dynamic image analysis will occur tothose skilled in the art.

Software Elements of the Present Invention

FIG. 14 shows the organization of a set of software elements (e.g.,program instructions of a computer readable medium) used by anillustrative embodiment to analyze frames, make judgments, sense inputs,and control output signals. The elements may be implemented using aclass hierarchy in a conventional object-oriented programming languagesuch as C++, so that each of the elements corresponds to a class.However, any acceptable programming technique and/or language can beused to carry out the processes described herein.

As illustrated, classes with a dotted border, such as Gadget class 1400,are abstract base classes that do not exist by themselves but are usedto build concrete derived classes such as Locator class 1420. Classeswith a solid border represent dynamic objects that can be created anddestroyed as needed by the user in setting up an application, using anHMI 830. Classes with a dashed border, such as Input class 1450,represent static objects associated with specific hardware or softwareresources. Static objects always exist and cannot be created ordestroyed by the user.

All classes are derived from Gadget class 1400, and so all objects thatare instances of the classes shown in FIG. 14 are a kind of Gadget. Inan illustrative embodiment, every Gadget:

-   -   1. has a name that can be chosen by the user;    -   2. has a logic output (a fuzzy logic value) that can be used as        a logic input by other gadgets to make judgments and control        output signals;    -   3. has a set of parameters than can be configured by a user to        specify its operation;    -   4. has one such parameter that can be used to invert the logic        output (i.e. fuzzy NOT); and    -   5. can be run, which causes its logic output to be updated based        on its parameters, logic inputs if any, and for certain Gadgets        the contents of the current frame, and which may also cause        side-effects such as the setting of an output signal.

The act of analyzing a frame consists of running each Gadget once, in anorder determined to guarantee that all logic inputs to a Gadget havebeen updated before the Gadget is run. In some embodiments, a Gadget isnot run during a frame where its logic output is not needed.

The Photo class 1410 is the base class for all Gadgets whose logicoutput depends on the contents of the current frame. These are theclasses that actually do the image analysis. Every Photo measures somecharacteristic of a region of interest (ROI) of the current frame. TheROI corresponds to a visible feature on the object to be inspected. Thismeasurement is called the Photo's analog output. The Photo's logicoutput is computed from the analog output by means of a fuzzy threshold,called the sensitivity threshold, that is among its set of parametersthat can be configured by a user. The logic output of a Photo can beused to provide evidence to be used in making judgments.

The Detector class 1430 is the base class for Photos whose primarypurpose is to make measurements in an ROI and provide evidence to beused in making judgments. In an illustrative embodiment all DetectorROIs are circles. A circular ROI simplifies the implementation becausethere is no need to deal with rotation, and having only one ROI shapesimplifies what the user has to learn. Detector parameters include theposition and diameter of the ROI.

A Brightness Detector 1440 measures a weighted average or percentilebrightness in the ROI. A Contrast Detector 1442 measures contrast in theROI. An Edge Detector 1444 measures the extent to which the ROI lookslike an edge in a specific direction. A Spot Detector 1446 measures theextent to which the ROI looks like a round feature such as a hole. ATemplate Detector 1448 measures the extent to which the ROI looks like apre-trained pattern selected by a user. The operation of the Detectorsis further described below.

The Locator class 1420 represents Photos that have two primary purposes.The first is to produce a logic output that can provide evidence formaking judgments, and in this they can be used like any Detector. Thesecond is to determine the location of an object in the field of view ofa vision detector, so that the position of the ROI of other Photos canbe moved so as to track the position of the object. Any Locator can beused for either or both purposes.

In an illustrative embodiment, a Locator searches a one-dimensionalrange in a frame for an edge. The search direction is normal to theedge, and is among the parameters to be configured by the user. Theanalog output of a Locator is similar to that for an Edge Detector.Locators are further described below.

The Input class 1450 represents input signals to the vision detector,such as an external trigger. The Output class 1452 represents outputsignals from the vision detector, such as might be used to control areject actuator. There is one static instance of the Input class foreach physical input, such as exemplary input signal 926 (FIG. 9), andone static instance of the Output class for each physical output, suchas exemplary output signals 922 and 924.

The Gate base class 1460 implements fuzzy logic decision making. EachGate has one or more logic inputs than can be connected to the logicoutputs of other Gadgets. Each logic input can be inverted (fuzzy NOT)by means of a parameter that a user can configure. An AND Gate 1462implements a fuzzy AND operation, and an OR Gate 1464 implements a fuzzyOR operation.

The Judge class 1470 is the base class for two static objects, theObjectDetect Judge 1472 and the ObjectPass Judge 1474. Judges implementdynamic image analysis by weighing evidence over successive frames tomake the primary decisions. Each Judge has a logic input to which a userconnects the logic output of a Photo or, more typically, a Gate thatprovides a logical combination of Gadgets, usually Photos and otherGates.

The ObjectDetect Judge 1472 decides if an object has been detected, andthe ObjectPass Judge 1474 decides if it passes inspection. The logicinput to the ObjectDetect Judge provides the object detection weight foreach frame, and the logic input to the ObjectPass Judge provides theobject pass score for each frame.

The logic output of the ObjectDetect Judge provides a pulse thatindicates when a judgment has been made. In one mode of operation,called “output when processing”, the leading edge of the pulse occurswhen the inspection of an object begins, for example at the end ofanalysis step 540 in FIG. 5, and the trailing edge occurs when theinspection of an object is complete, for example at the end of analysisstep 548. In another mode, called “output when done”, the leading edgeof the pulse occurs when the inspection of an object is complete, forexample at the end of analysis step 548 in FIG. 5, and the trailing edgeoccurs some time after that, for example at the end of idle step 580.

The logic output of the ObjectPass Judge provides a level that indicateswhether the most recently inspected object passed. The level changesstate when the inspection of an object is complete, for example at theend of analysis step 548.

FIG. 15 shows an example of how Photos can be used to inspect an object.FIG. 15 represents an image of object 110 (from FIG. 1), containinglabel feature 120 and hole feature 124, with superimposed graphicsrepresenting the Photos, and is displayed on an HMI 830 for a user toview and manipulate. A display of an image and superimposed graphics onan HMI is called an image view.

FIG. 15 represents an image view, showing an object 1500 containing alabel 1510 and a hole 1512. The object in this example contains 6visible features to be inspected, corresponding to the two Locators andfour Detectors further described below.

A Locator 1520 is used to detect and locate the top edge of the object,and another Locator 1522 is used to detect and locate the right edge.

A Brightness Detector 1530 is used to help detect the presence of theobject. In this example the background is brighter than the object, andthe sensitivity threshold is set to distinguish the two brightnesslevels, with the logic output inverted to detect the darker object andnot the brighter background.

Together the Locators 1520 and 1522, and the Brightness Detector 1530,provide the evidence needed to judge that an object has been detected,as further described below.

A Contrast Detector 1540 is used to detect the presence of the hole1512. When the hole is absent the contrast would be very low, and whenpresent the contrast would be much higher. A Spot Detector could also beused.

An Edge Detector 1560 is used to detect the presence and position of thelabel 1510. If the label is absent, mis-positioned horizontally, orsignificantly rotated, the analog output of the Edge Detector would bevery low.

A Brightness Detector 1550 is used to verify that the correct label hasbeen applied. In this example, the correct label is white and incorrectlabels are darker colors.

As the object moves from left to right through the field of view of thevision detector, Locator 1522 tracks the right edge of the object andrepositions Brightness Detector 1530, Contrast Detector 1540, BrightnessDetector 1550, and Edge Detector 1560 to be at the correct positionrelative to the object. Locator 1520 corrects for any variation in thevertical position of the object in the field of view, repositioning thedetectors based on the location of the top edge of the object. Ingeneral Locators can be oriented in any direction.

A user can manipulate Photos in an image view by using well-known HMItechniques. A Photo can be selected by clicking with a mouse, and itsROI can be moved, resized, and rotated by dragging. Additionalmanipulations for Locators are described below.

FIG. 16 shows a logic view containing a wiring diagram corresponding tothe example setup of FIG. 15. A wiring diagram shows all Gadgets beingused to inspect objects and interface to automation equipment, and theconnections between logic inputs and outputs of the Gadgets. A wiringdiagram is displayed on an HMI 830 for a user to view and manipulate. Adisplay of gadgets and their logic interconnections on an HMI is calleda logic view.

Referring still to the wiring diagram of FIG. 16, a Locator 1620 named“Top”, corresponding to Locator 1520 in the image view of FIG. 15, isconnected to AND Gate 1610 by wire 1624. Similarly, “Side” Locator 1622,corresponding to Locator 1522, and “Box” Detector 1630, corresponding toBrightness Detector 1530, are also wired to AND Gate 1610. The logicoutput of “Box” detector 1630 is inverted, as shown by the small circle1632 and as described above to detect the darker object against alighter background.

The logic output of AND Gate 1610 represents the level of confidencethat the top edge of the object has been detected, the right edge of theobject has been detected, and the background has not been detected. Whenconfidence is high that all three conditions are true, confidence ishigh that the object itself has been detected. The logic output of ANDGate 1610 is wired to the ObjectDetect Judge 1600 to be used as theobject detection weight for each frame.

Since the logic input to the ObjectDetect Judge in this case depends onthe current frame, the vision detector is operating in visual eventdetection mode. To operate in external trigger mode, an Input Gadgetwould be wired to ObjectDetect. To operate in continuous analysis mode,nothing would be wired to ObjectDetect.

The choice of Gadgets to wire to ObjectDetect is made by a user based onknowledge of the application. In the example of FIGS. 15 and 16, a usermay have determined that detecting just the top and right edges was notsufficient to insure that an object is present. Note that Locator 1522might respond to the label's left edge just as strongly as the object'sright edge, and perhaps at this point in the production cycle Locator1520 might occasionally find some other edge in the background. Byadding Detector 1530, and requiring all three conditions by means of ANDGate 1610, object detection is made reliable.

In the wiring diagram, Contrast Detector “Hole” 1640, corresponding toContrast Detector 1540, Brightness Detector “Label” 1650, correspondingto Brightness Detector 1550, and Edge Detector “LabelEdge” 1660,corresponding to Edge Detector 1560, are wired to AND Gate 1612. Thelogic output of AND Gate 1612 represents the level of confidence thatall three image features have been detected, and is wired to ObjectPassJudge 1602 to provide the object pass score for each frame.

The logic output of ObjectDetect Judge 1600 is wired to AND Gate 1670.The logic output of ObjectPass Judge 1602 is inverted and also wired toAND Gate 1670. The ObjectDetect Judge is set to “output when done” mode,so a pulse appears on the logic output of ObjectDetect Judge 1600 afteran object has been detected and inspection is complete. Since the logicoutput of ObjectPass 1602 has been inverted, this pulse will appear onthe logic output of AND Gate 1670 only if the object has not passedinspection. The logic output of AND Gate 1670 is wired to an Outputgadget 1680, named “Reject”, which controls an output signal from thevision detector than can be connected directly to a reject actuator 170.The Output Gadget 1680 is configured by a user to perform theappropriate delay 570 needed by the downstream reject actuator.

A user can manipulate Gadgets in a logic view by using well-known HMItechniques. A Gadget can be selected by clicking with a mouse, itsposition can be moved by dragging, and wires can be created by adrag-drop operation.

To aid the user's understanding of the operation of the vision detector,Gadgets and/or wires can change their visual appearance to indicatefuzzy logic values. For example, Gadgets and/or wires can be displayedred when the logic value is below 0.5, and green otherwise. In FIG. 16,wires 1604 and 1672 are drawn with dashed lines to indicate a logicvalue below 0.5, and other wires, for example wire 1624, are drawn solidto indicate logic values equal to or greater than 0.5.

One skilled in the art will recognize that a wide variety of objects canbe detected and inspected by suitable choice, configuration, and wiringof Gadgets. One skilled in the art will also recognize that the Gadgetclass hierarchy is only one of many software techniques that could beused to practice the invention.

Image Analysis Methods for Detectors

FIG. 17 illustrates a method for implementing Brightness and ContrastDetectors. In one embodiment of a Brightness Detector, the analog outputis the average gray level within the ROI. In an illustrative embodiment,a kernel of positive weights 1700 is created corresponding to the sizeand shape of the ROI, and the analog output A is the weighted averagegray level

$\begin{matrix}{A = \frac{\sum\limits_{i}{w_{i}z_{i}}}{\sum\limits_{i}w_{i}}} & (7)\end{matrix}$

where w_(i) is the i^(th) weight and z_(i) is the corresponding pixelgray level. In the illustrative embodiment of FIG. 17, the weightsapproximate a Gaussian function of distance r from the center of thekernel to the center of each weight,

$\begin{matrix}{{w(r)} = {a\;{\mathbb{e}}^{{- \frac{1}{2}}{(\frac{r}{\sigma})}^{2}}}} & (8)\end{matrix}$

so that pixels near the center are weighted somewhat higher than thosenear the edge. One advantage of a center-weighted Brightness Detector isthat if a bright feature happens to lie near the edge of the Detector'sROI, then slight variations in its position will not cause largevariations in the analog output. In FIG. 17 a=99, but any suitable valuecan be used. The value σ is set based on the diameter d of the kernel,

$\begin{matrix}{\sigma = {b\frac{\left( {d - 1} \right)}{2}}} & (9)\end{matrix}$

In the illustrative embodiment of FIG. 17, b=1.0.

In another illustrative embodiment, the analog output is defined by thefunction C(q), which is the gray level such that:

$\begin{matrix}{\frac{\sum\limits_{k❘{z_{k} \leq {C{(q)}}}}w_{k}}{\sum\limits_{i}w_{i}} = q} & (10)\end{matrix}$

where q is a percentile chosen by a user. C is the inverse cumulativeweighted distribution of gray levels. Various useful values of q aregiven in the following table:

q C(q) 0.0 absolute minimum gray level in ROI 0.1 statistically reliableminimum gray level 0.5 weighted median gray level 0.9 statisticallyreliable maximum gray level 1.0 absolute maximum gray level

In one embodiment of a Contrast Detector, the analog output is thestandard deviation of the gray levels within the ROI. In an illustrativeembodiment, the array of positive weights 1700 is used to compute aweighted standard deviation:

$\begin{matrix}{A = \frac{\sqrt{{\sum\limits_{i}{w_{i}{\sum\limits_{i}{w_{i}z_{i}^{2}}}}} - \left( {\sum\limits_{i}{w_{i}z_{i}}} \right)^{2}}}{\sum\limits_{i}w_{i}}} & (11)\end{matrix}$

In another illustrative embodiment, the analog output is given byC(q _(hi))−C(q _(lo))  (12)

where the q values may be chosen by the user. Useful values areq_(hi)=0.95, q_(lo)=0.05.

FIG. 18 illustrates a method for implementing an Edge Detector to detectstep edges. A step kernel 1800 is created corresponding to the size andshape of the ROI, and the intended direction of the edge. For stepkernel 1800, the ROI is a circle 12 pixels in diameter, and thedirection of the edge is 15 degrees from horizontal. The step kernel1800 is an approximation of the first derivative of a Gaussian functionof distance t from the edge to the center of each weight,

$\begin{matrix}{{w(r)} = {\frac{ar}{\sigma}{\mathbb{e}}^{- {\frac{1}{2}{\lbrack{{(\frac{r}{\sigma})}^{2} - 1}\rbrack}}}}} & (13)\end{matrix}$

In FIG. 18 a=99, but any suitable value can be used. In an illustrativeembodiment, equation 9 is used with b=0.5.

The step kernel 1800, with values k_(i), can be considered to be theproduct of an ideal step edge template e_(i) and a kernel of positiveweights w_(i):

$\begin{matrix}{{w_{i} = {k_{i}}}{e_{i} = \frac{k_{i}}{w_{i}}}{k_{i} = {e_{i}w_{i}}}} & (14)\end{matrix}$

Note that the ideal step edge template values e_(i) are +1 when k_(i)>0,corresponding to the black on white region of step kernel 1800, and −1when k_(i)<0, corresponding to the white on black region of step kernel1800.

Define contrast C and weighted normalized correlation R² of the stepkernel and a like-shaped ROI with pixel values z_(i) as follows:

$\begin{matrix}{{v = {{\sum\limits_{i}{w_{i}{\sum\limits_{i}{w_{i}z_{i}^{2}}}}} - \left( {\sum\limits_{i}{w_{i}z_{i}}} \right)^{2}}}{C = \frac{\sqrt{v}}{\sum\limits_{i}w_{i}}}{R^{2} = \frac{\left( {\sum\limits_{i}{k_{i}z_{i}}} \right)^{2}}{v}}} & (15)\end{matrix}$

The contrast C uses the standard formula for weighted standarddeviation, and R² uses the standard formula for weighted normalizedcorrelation, but simplified because for step kernel 1800

$\begin{matrix}{{{\sum\limits_{i}{w_{i}e_{i}}} = {{\sum\limits_{i}k_{i}} = 0}}{{\sum\limits_{i}{w_{i}e_{i}^{2}}} = {\sum\limits_{i}w_{i}}}} & (16)\end{matrix}$

An orthogonal step kernel 1810 with values k_(i)′ is also created thatis identical to the step kernel 1800 but rotated 90 degrees. The ratio

$\begin{matrix}{D = {\frac{\sum\limits_{i}{k_{i}^{\prime}z_{i}}}{\sum\limits_{i}{k_{i}z_{i}}}}} & (17)\end{matrix}$

is a reasonable estimate of the tangent of the angle between the actualand expected direction of an edge, particularly for small angles where Dis also a good estimate of the angle itself. Note that an orthogonalstep template 1810 doesn't need to be created—the values from the steptemplate 1800 can be used, but corresponding to the pixels values in theROI in a different order.

FIG. 19 shows how the values R2, C, and D are used to determine theanalog output of an illustrative embodiment of the Edge Detector. Onecan be confident that an edge has been detected when three conditionsare met:

-   -   1. The ROI looks like an ideal step edge, which means that the        weighted normalized correlation R² of the ideal step edge        template and the ROI is high;    -   2. The contrast C is significantly above some noise threshold;        and    -   3. The angle D is small.

A weighted normalized correlation operation 1900 using ROI 1910 and stepkernel 1920 computes R². A contrast operation 1930 using ROI 1910 andstep kernel 1920 computes C, which is converted by fuzzy thresholdoperation 1940 into a fuzzy logic value 1942 indicating the confidencethat the contrast is above the noise level. Weighted correlationoperations 1950 and 1952, using ROI 1910, step kernel 1920, andorthogonal step kernel 1922, and absolute value of arctangent of ratiooperation 1960, compute D, which is converted by fuzzy thresholdoperation 1970 into a fuzzy logic value 1972 indicating the confidencethat the angle between the expected and actual edge directions is small.

A fuzzy AND element 1980 operates on R² and fuzzy logic values 1942 and1972 to produce the analog output 1990 of the Edge Detector. Note thatR², being in the range 0-1, can be used directly as a fuzzy logic value.The analog output 1990 is in the range 0-1, but it can be multiplied bysome constant, for example 100, if a different range is desired. Notethat the logic output of an Edge Detector is derived from the analogoutput using the sensitivity threshold that all Photos have.

FIG. 20 illustrates a method for implementing an Edge Detector to detectridge edges. A ridge kernel 2000 is created corresponding to the sizeand shape of the ROI, and the intended direction θ of the edge. Forridge kernel 2000, the ROI is a circle 12 pixels in diameter, and thedirection θ is 15 degrees from horizontal. The ridge kernel 2000 is anapproximation of the second derivative of a Gaussian function ofdistance r from the edge to the center of each weight,

$\begin{matrix}{{w(r)} = {{a\left\lbrack {1 - \left( \frac{r}{\sigma} \right)^{2}} \right\rbrack}{\mathbb{e}}^{{- \frac{1}{2}}{(\frac{r}{\sigma})}^{2}}}} & (18)\end{matrix}$

In FIG. 20 a=99, but any suitable value can be used. In an illustrativeembodiment, equation 9 is used with b=0.33.

The use of ridge kernel 2000 is similar to that for step kernel 1800.The contrast C is computed using the same formula, but R² uses adifferent formula because the sum of the kernel values is not 0:

$\begin{matrix}{R^{2} = \frac{\left( {{\sum\limits_{i}{w_{i}{\sum\limits_{i}{k_{i}z_{i}}}}} - {\sum\limits_{i}{w_{i}z_{i}{\sum\limits_{i}k_{i}}}}} \right)^{2}}{v\left\lbrack {\left( {\sum\limits_{i}w_{i}} \right)^{2} - \left( {\sum\limits_{i}k_{i}} \right)^{2}} \right\rbrack}} & (19)\end{matrix}$

Note that this formula reduces to the one used for step edges when thesum of the kernel values is 0.

A different method is used to determine the angle D between the actualand expected edge directions. A positive rotated ridge kernel 2020 withvalues k_(i) ⁺ is created with an edge direction θ+a, and a negativerotated ridge kernel 2010 with values k_(i) ⁻ is created with an edgedirection θ−a. A parabola is fit to the three points

$\begin{matrix}{\left( {0,{\sum\limits_{i}{k_{i}z_{i}}}} \right)\left( {a,{\sum\limits_{i}k_{i}^{+}}} \right)\text{}\left( {{- a},{\sum\limits_{i}k_{i}^{-}}} \right)} & (20)\end{matrix}$

The x coordinate of the minimum of the parabola is a good estimate ofthe angle D between the actual and expected edge directions.

FIG. 21 shows how the ridge kernels are used to determine the analogoutput of an illustrative embodiment of an Edge Detector that can detecteither step or ridge edges. For ridge edge detection, weightednormalized correlation 2100 uses ROI 2110 and ridge kernel 2120 tocompute R². Contrast 2130 uses ROI 2110 and ridge kernel 2120 to computeC, which is then converted to a fuzzy logic value by fuzzy threshold2140. Correlation elements 2150, 2152, and 2154 use ROI 2110 and ridgekernel 2120, positive rotated ridge kernel 2124, and negative rotatedridge kernel 2122 to provide input to parabolic fit 2160 to computeangle D, which is then converted to a fuzzy logic value by fuzzythreshold 2170.

R² and the fuzzy logic values are used by fuzzy AND element 2180 toproduce a ridge analog output 2192 for an Edge Detector that can detectridge edges. For an Edge Detector that can detect either step or ridgeedges, the ridge analog output 2192 and analog output 1990 from a stepedge detector 2188 can be used by fuzzy OR element 2182 to produce acombined analog output 2190.

FIG. 22 shows graphical controls that can be displayed on an HMI for auser to view and manipulate in order to set parameters for an EdgeDetector. A set of graphical controls displayed on HMI 830 for settingGadget parameters is called a parameter view. The parameter view forother Photos is similar enough to FIG. 22 that it would be obvious toone skilled in the art how to construct them.

Name text box 2200 allows a user to view and enter a Gadget's name. Timelabel 2202 shows the time taken by the most recent run of a Gadget.Logic output label 2204 shows a Gadget's current logic output value, andmay change color, shape, or other characteristic to distinguish betweentrue (≧0.5) and false (<0.5). Invert checkbox 2206 allows a Gadget'slogic output to be inverted.

Thumbs-up button 2210 and thumbs-down button 2212 are used for learning,as further described below (FIG. 45).

Position controls 2220 are used to position a Photo in the field ofview. Diameter spinner 2222 is used to change the diameter of aDetector. Direction controls 2224 are used to orient an Edge Detector tothe expected edge direction. Position, diameter, and orientation canalso be set by manipulation of graphics in an image view, for examplethe image view of FIG. 15.

Edge type checkboxes 2230 are used to select the types of edges to bedetected and the edge polarity. Dark-to-light step, light-to-dark step,dark ridge, and light ridge can be selected. Any combination of choicesis allowed, except for choosing none.

Jiggle spinner 2240 allows the user to specify a parameter j such thatthe Edge Detector will be run at a set of positions ±j pixels around thespecified position, and the position with the highest analog output willbe used.

Sensitivity threshold controls 2250 allow the user to set thesensitivity fuzzy threshold of a Photo. Zero-point label 2251 showsvalue t₀ 1120 (FIG. 11), which can be set by zero-point slider 2252.One-point label 2253 shows value t₁ 1122, which can be set by one-pointslider 2254. Analog output label 2255 shows the current analog output ofa Photo. The analog output is also shown graphically by the filled-inregion to the left of analog output label 2255, which shrinks and growslike a mercury thermometer lying on its side. The filled-in region canbe displayed in three distinct colors or patterns corresponding to afirst zone 2256 below t₀, a second zone 2257 between t₀ and t_(i), and athird zone 2258 above t₁.

Contrast threshold controls 2260 allow the user to view the contrast Cand set the contrast fuzzy thresholds 1940 and 2140. These controlsoperate in the same manner as the sensitivity threshold controls 2250.

Direction error controls 2270 allow the user to view the angle betweenthe actual and expected edge directions D and set the direction fuzzythresholds 1970 and 2170. These controls operate in the same manner asthe sensitivity threshold controls 2250, except that the thermometerdisplay fills from right-to left instead of left-to-right because lowervalues of D correspond to higher fuzzy logic values.

FIG. 23 illustrates a method for implementing a Spot Detector. A spotkernel 2300 is created corresponding to the size and shape of the ROI.For spot kernel 2300, the ROI is a circle 15 pixels in diameter. Thespot kernel 2300 is an approximation of the second derivative of aGaussian function of distance r from the center of the kernel to thecenter of each weight, using equations 18 and 9. In an illustrativeembodiment, b=0.6.

The use of spot kernel 2300 is similar to that for ridge kernel 2000.Weighted normalized correlation R² and contrast C are computed using thesame formulas as was used for the ridge kernel.

FIG. 24 shows how the spot kernel is used to determine the analog outputof an illustrative embodiment of a Spot Detector. Operation of the SpotDetector is identical to the Edge Detector embodiment shown in FIG. 19,except that angle D is not computed or used. A weighted normalizedcorrelation 2400 uses ROI 2410 and spot kernel 2420 to compute R².Contrast 2430 uses ROI 2410 and spot kernel 2420 to compute C, which isthen converted to a fuzzy logic value by fuzzy threshold 2440. R² andthe fuzzy logic value are used by fuzzy AND element 2480 to produce aspot analog output 2490.

Methods and Human-Machine Interface for Locators

FIG. 25 shows a pair of image views that will be used to describe theoperation of Locators according to an illustrative embodiment. In afirst image view 2500 and a second image view 2502 there is one Detector2510 and one Locator 2512. The reader will understand that the followingdescription of Detector 2510 and Locator 2512 applies generally to anyDetector and Locator. The reader will further understand that manyalternate methods can be devised for configuring Locators within thescope of the invention.

In an illustrative embodiment, a Locator searches a one-dimensionalrange for an edge, using any of a variety of well-known techniques. Thesearch direction is normal to the edge, and a Locator has a widthparameter that is used to specify smoothing along the edge, which isused in well-known ways. The analog output of a Locator depends on theparticular method used to search for the edge.

In an illustrative embodiment, a Locator searches a one-dimensionalrange for an edge using the well-known method of computing a projectionof the ROI parallel to the edge, producing a one-dimensional profilealong the search range. The one-dimensional profile is convolved with aone-dimensional edge kernel, and the location of the peak responsecorresponds to the location of the edge. A interpolation, such as thewell-known parabolic interpolation, can be used if desired to improvethe edge location accuracy. In another embodiment, an edge can belocated by searching for a peak analog output using the edge detector ofFIG. 19 or FIG. 21, once again interpolating to improve accuracy ifdesired.

In another embodiment, a Locator searches a multi-dimensional range,using well-known methods, that may include translation, rotation, andsize degrees of freedom. It will be clear to one skilled in the art howto employ multi-dimensional Locators to position Photos in practicingthe invention, so the following discussion will be limited toone-dimensional Locators, which are preferred due to their simplicity.

Detector 2510 and Locator 2512 can be moved around in the FOV byclicking anywhere on their border and dragging. Detector 2510 has aresize handle 2520 for changing its diameter, and Locator 2512 has aresize handle 2522 for changing its width and range, and a rotate handle2524 for changing its direction. All Photos can be moved by dragging theborder, and have similar handles as appropriate to their operation.

In the illustrative embodiment of FIG. 25, a Locator is drawn in animage view as a rectangle with a inside line segment called the plunger2530. The width of the Locator is along the plunger, and its range isnormal to the plunger. A locator is oriented by a user so that theplunger is approximately parallel to the edge to be found. The rectangleshows the search range, and the plunger shows the location of a detectededge, if any. If no edge is detected, the plunger is drawn in the centerof the range.

A Locator has a rail 2532, shown in the Figure as a dashed line, that iscoincident with the plunger but extending in both directions to the edgeof the image view.

Every Photo can be linked to zero or more locators, up to some maximumnumber determined by the particular embodiment of the invention. Thenumber of links determines the number of degrees of freedom that theLocators can control. Degrees of freedom include rotation, size, and thetwo degrees of freedom of translation. In an illustrative embodiment,the maximum number of links is two and only the translation degrees offreedom are controlled.

A linkage defines how a Photo moves as the Locator's plunger moves,following an edge in the image. The movements are defined to keep thePhoto at a constant relative distance to the rail or rails of thelocators to which it is linked. In an illustrative embodiment thelinkages are drawn using a mechanical analogy, such that one couldactually build a linkage out of structural elements and bearings and thePhotos would move in the same way as forces are applied to the plungers.

In FIG. 25 the linkage from Detector 2510 to Locator 2512 includes a rod2540, which is rigidly attached to Detector 2510 by a post 2542, and toa slider 2544 that is free to move along the rail 2532, but which holdsthe rod at right angles to the rail. The post is drawn on the border ofa Photo such that the rod, if extended, would pass through the center ofthe Photo and at the closest of the two possible such points to therail. A Locator's rail is only shown if there are linkages to it.

Every photo has an emitter, a diamond-shaped handle drawn somewhere onthe border. For example Detector 2510 has emitter 2550 and Locator 2512has emitter 2552. A link is created by drag-dropping a Photo's emitterto any point on a Locator. If the link already exists, the drag-dropmight delete the link, or another mechanism for deleting might be used.The user may not create more than the maximum number of allowable linksfrom any Photo, nor any circular dependencies. To aid the user during anemitter drag over a Locator, a tool tip can be provided to tell the userwhether a link would be created, deleted, or rejected (and why).

Dragging a Locator does not change the behavior of its plunger—it stayslocked on an edge if it can find one, or reverts to the center if not.Thus dragging a locator while an edge is detected just changes itssearch range; the plunger does not move relative to the FOV. Moregenerally, dragging a Locator never changes the position of any Photo towhich it is linked. Dragging a Locator will adjust the rod lengths asnecessary to insure that no other Photo moves relative to the FOV.

Any plunger may be dragged manually within the range of its Locator,whether or not it has found an edge, and any linked Photos will moveaccordingly. This allows users to see the effect of the linkages. Assoon as the mouse button is released, the plunger will snap back to itsproper position (moving linked Photos back as appropriate).

In FIG. 25 Detector 2510 is linked to one Locator 2512, and so onetranslation degree of freedom is controlled. The degree of freedom isnormal to the edge direction, which means that it is in the direction ofrod 2540. Comparing second image view 2502 with first image view 2500,the plunger 2530 has moved to the right as it follows an edge (notshown) in the image. Note that the position in the FOV of Locator 2512has not changed, but Detector 2510 has moved to the right to follow theplunger, which is following an edge of an object and therefore followingthe motion of the object itself. In our mechanical analogy, Detector2510 moves because it is rigidly attached to rail 2532 by rod 2540, andthe rail moves with the plunger.

FIG. 26 shows a pair of image views that will be used to explain thebehavior of a Detector linked to two Locators. In a first image view2600 and a second image view 2602 Detector 2610 is linked to a firstLocator 2620 and a second Locator 2630, and so two translation degreesof freedom are controlled. The degrees of freedom are in the directionof first rod 2622 and second rod 2632. Note that the two degrees offreedom are not independent because they are not orthogonal. Handles andemitters are not shown in FIG. 26.

Comparing second image view 2602 with first image view 2600, firstplunger 2624 has moved down as it follows a first edge (not shown) inthe image, and second plunger 2634 has moved to the left and slightlydown as it follows a second edge (not shown). Note that the positions inthe FOV of Locators 2620 and 2630 have not changed, but Detector 2610has moved down and to the left to follow the plungers, which isfollowing the edges of an object and therefore following the motion ofthe object itself.

In our mechanical analogy, Detector 2610 moves because it is rigidlyattached to first rail 2626 by first rod 2622, and to second rail 2636by second rod 2632. Note that first slider 2628 has slid to the leftalong first rail 2626, and second slider 2638 has slid down along secondrail 2636. The sliders slide along the rails when two non-orthogonalLocators are linked to a Photo.

If a Photo is linked to two nearly parallel Locators, its motion wouldbe unstable. It is useful to set an angle limit between the Locators,below which the linked Photo will not be moved. This state can beindicated in some way in the image view, such as by displaying the tworods using a special color such as red.

The ability to have Locators either at fixed positions or linked toother Locators provides important flexibility. In FIG. 26 neitherLocator is linked and so they remain at fixed positions in the FOV, andtherefore at fixed positions relative to the illumination, which isoften desirable.

FIG. 27 shows a pair of image views that will be used to explain thebehavior of a Detector linked to two Locators, where one of the Locatorsin linked to the other. In a first image view 2700 and a second imageview 2702 Detector 2710 is linked to a first Locator 2720 and a secondLocator 2730. Second Locator 2730 is also linked to first Locator 2720via rod 2740, post 2742, and slider 2744. Slider 2744 slides along rail2722 of first locator 2720.

Note that there need be no limit on the number of Photos that can belinked to a Locator; the degree of freedom limit is on the number oflinks one Photo can have to Locators. In the example of FIG. 27,Detector 2710 is linked to two Locators and is controlled in twotranslation degrees of freedom. Second Locator 2730 is linked to oneLocator and is controlled in one translation degree of freedom. FirstLocator 2720 is linked to no Locators and remains fixed in the FOV.

The Locators are configured to follow the top and right edges of acircular feature 2750. Comparing second image view 2702 with first imageview 2700, the circular feature 2750 has moved down, causing rail 2722to move down to follow it. This moves both Detector 2710 and secondLocator 2730 down. Note that Detector 2710 is at the same positionrelative to the object, and so is second Locator 2730. This is desirablein this case, because if second Locator 2730 were fixed in the FOV, itmight miss the right edge of circular feature 2750 as it moves up anddown. Note that this would not be problematic if the edge of an objectin the image was a straight line.

First Locator 2720 has no Locator to move it left and right so as tofind the top edge of circular feature 2750. It can't link to secondLocator 2730 because that would create a circular chain of links, whichis not allowed because one Locator has to run first and it can't belinked to anything. Instead, the motion of the object through the FOVinsures that first Locator 2720 will find the top edge. In the exampleof FIG. 27 the motion is left to right, and due to the high frame rateof a vision detector the object moves only slightly each frame.Eventually, first Locator 2720 will find the top edge, and will do so ona number of frames, depending on the speed of the object, where the topof circular feature 2750 is close to the center of the Locator. On thoseframes, second Locator 2730 will be positioned properly to find theright edge, and it will move Detector 2710 left and right as needed tokeep it in the right position.

FIG. 28 shows a method for handling cases where the edge to be found bya Locator does not extend in a straight line, and so placement of thelocator must be fairly precise along the object boundary. This methodcould be used for first Locator 2720 in FIG. 27 in an application whereobjects move at very high speed, and so there might be a chance ofmissing the top edge entirely as the object moves through the FOV.

To handle cases like this, Locators have a parameter that can be used tospecify the number of parallel sweeps to be made in searching for theedge. The sweeps are spaced apart along the edge by an amount thatprovides sufficient overlap so that the edge won't fall between thecracks of the sweeps.

FIG. 28 shows a Locator 2800 with four sweeps that has found an edge onthe second sweep from the left. Triangular-shaped sweep markers,including example sweep markers 2810 and 2812, are shown outside thedashed sweep rectangle 2820 to avoid interference from the locatorgraphics within. If an edge is not found on any of the sweeps, theLocator reverts to the center of the sweep rectangle (which won't be ata sweep marker for even sweep counts).

FIG. 29 shows how Locators can be used to handle object rotation andsize change even in embodiments where only two translation degrees offreedom are controlled. The restriction to translation only providesconsiderable simplicity and transparency for the user, but small objectrotation and size changes can still be handled since Photos in differentparts of the FOV can translate differently in response to differentLocators. Small rotations and size changes are well-approximated bytranslations within a small region of the FOV, so as long as Photos arelinked to at least one nearby Locator, object rotation and size changewill look like translation.

Referring still to FIG. 29, a first image view 2900 and a second imageview 2902 contain a first Detector 2910, a second Detector 2912, a firstLocator 2920, a second Locator 2922, and a third Locator 2924.

First Detector 2910 is linked to nearby first Locator 2920 and secondLocator 2922, and will be positioned properly even if the object rotatesor changes size (as long as the change is not too big). But secondDetector 2912 is too far away—a rotation would tend to mis-positionsecond Detector 2912 vertically relative to second Locator 2922, and asize change would tend to mis-position it horizontally relative to firstLocator 2920. Third Locator 2924 is used instead of second Locator 2922to get the vertical position of second Detector 2912, allowing overallobject rotation to be handled. The remote first Locator 2920 is used toget horizontal position for second Detector 2912, so the object sizeshould not vary much. If size variation needs to be handled in additionto rotation, one would add a fourth Locator, near second Detector 2912and oriented horizontally.

Comparing second image view 2902 with first image view 2900, the object(not shown) has moved to the right and rotated counterclockwise, whichcan be seen by the motion of the Detectors as the Locators follow theobject edges. Note that second Locator 2922 and third Locator 2924 arelinked to first Locator 2920 so that they stay close to the Detectors.

Alternate Logic View using Ladder Diagrams

FIG. 30 shows an alternative representation for a logic view, based on aladder diagram, a widely used industrial programming language. Theladder diagram of FIG. 30 is essentially equivalent to the wiringdiagram shown in FIG. 16, in that it represents the same set of Gadgetsand their logic interconnections. In an illustrative embodiment, a usercan switch the logic view between a wiring diagram and a ladder diagramat will, and can manipulate and edit either diagram.

In an illustrative embodiment, to render a ladder diagram for aconfiguration of Gadgets one rung is created for each Gadget that haslogic inputs: Gates, Judges, and Outputs. The order of the rungs is theautomatically determined run order for the Gadgets. Each rung consistsof one contact for each logic input, followed by an icon for the Gadget.Contacts are normally open for non-inverted connections and normallyclosed for inverted connections. For AND Gates the contacts are inseries, and for OR Gates the contacts are in parallel. Labels associatedwith each contact indicate the name of the Gadget that is connected tothe logic input.

To simplify the ladder diagram, the user may choose to hide any Gatewhose logic output is connected to exactly one logic input of anotherGadget via a non-inverted connection. When a Gate is hidden, the rungfor that Gate is not displayed. Instead the contacts for that Gatereplace the normally open contact that would have appeared on the onerung in which that Gate is used.

Referring to FIG. 30, rung 3000 shows the inputs to ObjectDetect Judge3002, rung 3010 shows the inputs to ObjectPass Judge 3012, and rung 3020shows the inputs to Output Gadget 3022 named “Reject”. Normally opencontacts Top 3030 and Side 3032, and normally closed contact Box 3034,represent the inputs to AND Gate 1610 (FIG. 16), which has replaced thecontact that would have appeared on rung 3000 because AND Gate 1610 isconnected to exactly one logic input via a non-inverted connection.

Similarly, rung 3010 shows normally open contacts Hole 3040, Label 3042,and LabelEdge 3044 connected to an AND Gate (hidden AND Gate 1612),whose output is connected to ObjectPass Judge 3012.

Rung 3020 shows normally open contact 3050 and normally closed contact3052 connected to an AND Gate (hidden AND Gate 1670), whose output isconnected to Output Gadget 3022 named “Reject”.

Note that the ladder diagrams used by the above-described embodiment ofthe invention are a limited subset of ladder diagrams widely used inindustry. They are provided primarily to aid users who are familiar withladder diagrams. The subset is chosen to match the capabilities ofwiring diagrams, which simplifies the implementation and also allows auser to choose between wiring and ladder diagrams as need arises.

In another embodiment, a more complete implementation of ladder diagramsis provided and wiring diagrams are not used. The result combines thecapabilities of a vision detector with those of a PLC, resulting in apowerful industrial inspection machine but at a cost of increasedcomplexity.

Marking, Synchronized Outputs, and Related Measurements

FIG. 31 shows a timing diagram that will be used to explain how visiondetector output signals may be synchronized with the mark time. Signalsynchronization is desirable for a variety of industrial inspectionpurposes, such as control of a downstream reject actuator. Therequirements for these signals depends on how objects are presented andhow they are detected.

As discussed above, objects are detected either by visual eventdetection or external trigger (continuous analysis mode is used whenthere are no discrete objects). Furthermore, objects may be presentedeither by indexing, in which case they come to a stop in the FOV, orcontinuous motion. When using an external trigger the analysis istypically the same regardless of how objects are presented. For visualevent detection, however, the analysis may depend on whether the objectwill come to a stop (indexing) or be in roughly uniform motion(continuous). For example, a vision detector may not be able to measureor use object speed in an indexed application.

Visual event detection is a novel capability and suggests novel outputsignal control, particularly when continuous object presentation isused. It is desirable that a vision detector be able to control someexternal actuator, either directly or by serving as input to a PLC. Thissuggests, for continuous presentation at least, that the timing ofoutput signals be related with reasonable precision to the point in timewhen the object passed a particular, fixed point in the production flow.In the example of FIG. 4 the fixed point is mark point 430, and in thetimelines of FIGS. 5 and 6 the time is mark times 550, 552, and 680. InFIG. 31, the time is mark time 3100. Note that an encoder count may beused instead of time.

For prior art vision systems this goal is addressed by the externaltrigger, which is typically a photodetector that responds withinmicroseconds of the mark time. This signal, which triggers the visionsystem (e.g. signal 166 in FIG. 1), is also used by a PLC (signal 162)to synchronize vision system outputs with a downstream actuator. Thevision system outputs come many milliseconds later, at a time that isfar too variable to be used without synchronization.

The present invention, when used in visual event detection mode, canprovide outputs synchronized to reasonable precision with the mark time,whether it controls the actuator directly or is used by a PLC. Oneconcern, however, is that like a vision system and unlike aphotodetector, a vision detector makes its decision about the objectmany milliseconds after the mark time. Furthermore, the delay may bequite variable, depending on how many frames were analyzed and, to alesser extent, when in the acquire/process cycle the mark time occurs.

FIG. 31 shows the ObjectDetect logic output 3140 and the ObjectPasslogic output 3150. Note that the ObjectDetect Judge is in “output whendone” mode, which is the mode used for output signal synchronization.ObjectPass logic output 3150 changes state, and a detect pulse 3170appears on ObjectDetect logic output 3140, when the decision is made atdecision point 3110. Note that the decision delay 3130 from mark time3100 to the decision point 3110 may be variable.

If ObjectDetect logic output 3140 and ObjectPass logic output 3150 arewired to an AND Gate, a pulse (not shown) will be produced only when aobject that passes inspection is detected. If the logic output ofObjectPass is inverted, the AND Gate will produce a pulse (not shown)only when a object that fails inspection is detected.

The detect pulse 3170, and pulses indicating passing object detected andfailing object detected, are all useful. In an indexed application theymight be used directly by actuators. A PLC can use an external triggerto synchronize these pulses with actuators. But when objects are incontinuous motion and no external trigger is used, the pulses oftencannot be used directly to control actuators because of the variabledecision delay 3130.

The invention solves this problem by measuring the mark time 3100 andthen synchronizing an output pulse 3180 on output signal 3160 to it. Theoutput pulse 3180 occurs at a fixed output delay 3120 from mark time3100. Referring also to the timing diagram in FIG. 5, output pulse 3180is an example of a report step 560, and output delay 3120 corresponds todelay 570.

The act of measuring the mark time is called marking. The mark time canbe determined to sub-millisecond accuracy by linear interpolation,least-squares fit, or other well-known methods, using the known times(counts) at which the images were captured and the known positions ofthe object as determined by appropriate Locators. Accuracy will dependon shutter time, overall acquisition/processing cycle time, and objectspeed.

In an illustrative embodiment a user chooses one Locator whose searchrange is substantially along the direction of motion to be used formarking. The mark point is arbitrarily chosen to be the center point ofthe Locator's range—as discussed above, the mark point is an imaginaryreference point whose exact position doesn't matter as long as it isfixed. The user can achieve the desired synchronization of outputsignals by adjusting the delay from this arbitrary time. If an object isdetected that does not cross the mark point during the active frames,the mark time can be based on an extrapolation and the accuracy maysuffer.

The user may as an alternative specify that the mark time occurs whenthe object is first detected. This option might be selected inapplications where Locators are not being used, for example when visualevent detection relies on Detectors placed in fixed positions in the FOV(see FIG. 44). Marking at the point of first detection might be lessaccurate than when locators are used, although in some applications itmay be possible to improve accuracy by interpolating the fuzzy logicvalues input to the ObjectDetect Judge to find the time at which thevalue crosses 0.5.

Note that output signals can only be synchronized to the mark time ifoutput delay 3120 is longer than the longest expected decision delay3130. Thus the actuator should be sufficiently downstream of the markpoint, which is expected to be the case in almost all applications.

When an external trigger is used, the mark time is relative to the timeat which the trigger occurs (e.g. mark time 680 in FIG. 6). Output pulse3180 can be synchronized to this time for direct control of a downstreamactuator. When using an external trigger and a PLC, which is how priorart vision systems are used, marking can be used but is generally notnecessary.

FIG. 32 shows an example of measuring the mark time in visual eventdetection mode, and also measuring object speed, pixel size calibration,and object distance and attitude in space. Each column in the Figurecorresponds to one frame captured and analyzed by the invention, exceptfor the first column which serves to label the rows. Frame row 3210 is aframe number that serves to identify the frame. Time row 3220 shows thetime in milliseconds that the frame was captured, measured from somearbitrary reference time. Encoder row 3225 shows the encoder count atthe time the frame was captured. Object detect row 3230 shows the logicinput to the ObjectDetect Judge. From this it can be seen that theactive frames are 59-64.

Location row 3240 shows the location of an edge measured by a Locator,oriented to have a search range substantially in the direction of motionand chosen by a user. The location is measured relative to the center ofthe Locator's search range, and is shown only for the active frames. Itcan be seen that the location crosses zero, the mark point, somewherebetween frames 61 and 62, which is between times 44.2 and 46.2, andcounts 569 and 617.

In this example, dynamic image analysis ends on frame 66, after twoconsecutive inactive frames are found. A mark time based on location,shown in sixth row 3250, is computed at the end of frame 66. The valueshown, 45.0, is the linear interpolated time between frames 61 and 62where the location crosses zero. As an alternative, a line can be fit tothe points for the active frames from time row 3220 and location row3240, and that line can be used to calculate the time valuecorresponding to location 0.

A mark time based on the time that the object was first detected isshown in the seventh row 3260.

A considerable amount of additional useful information can be obtainedfrom the measured data, summarized in the following table:

two least-squares points fit mark time (ms) 45.00 45.01 mark count 588.2587.6 object speed (pixels/ms) 0.725 0.725 pixel size (counts/pixel)31.6 31.8

As can be seen, the information can be computed using two points (linearinterpolation using zero-crossing frames 61 and 62 for the mark, andslope using frames 59 and 64 for speed and size), or by a least-squaresfit, or by other methods known in the art. The results are similar forthe two methods shown, with the least-squares method generally beingmore accurate but more complex.

Mark time and count may be used for output signal synchronization asexplained above. Object speed may be communicated to automationequipment for a variety of purposes. The pixel size calculation gives acalibration of the size of a pixel in encoder counts, which areproportional to physical distance along the production line. Such acalibration can be used for a variety of well-known purposes, includingpresenting distances in the FOV in physical units for the user, andtransporting a setup between vision detectors that may have slightlydifferent optical magnifications by adjusting the size and position ofPhotos based on the calibrations.

Since pixel size can be calculated for every object, the value can beused to determine object distance. Smaller pixel sizes correspond toobjects that are farther away. For constant speed production lines thesame determination can be made using object speed, just as when lookingout a car window distant objects appear to be moving slower than nearbyones.

The data in FIG. 32 may be obtained from a single Locator designated bya user for that purpose. In another embodiment, a user could designatetwo locators separated either parallel or normal to the direction ofmotion in the FOV. All such Locators would be oriented to have theirsearch range parallel to the direction of motion. Differences in pixelsize calibration obtained from two such Locators represents objectrotation in space, about an axis normal to the direction of motion andin the plane of the object for Locators separated parallel to thedirection of motion, and about an axis parallel to the direction ofmotion for Locators separated normal to the direction of motion.

FIG. 33 shows an output FIFO that is used to generate synchronizedpulses for downstream control of actuators. In general many objects maylie between the inspection point and a downstream actuator, and a FIFOor similar mechanism is needed to hold information about the outputpulse, if any, corresponding to each such object until it reaches theactuator and the pulse is needed.

Referring to FIGS. 5 and 31, report step 560 for first object 500,corresponding to output pulse 3180, occurs after delay 570,corresponding to output delay 3120, which is after second object 510 hasbeen analyzed. Whenever output delay 3120 is greater than the timebetween objects, a FIFO or similar mechanism is needed.

FIG. 33 shows a simple FIFO that can be used for this purpose. The FIFOholds numbers corresponding to the time at which an output signal shouldchange state. At decision point 3110 (FIG. 31) an output pulse can bescheduled by placing times, for example leading edge time 3300 andtrailing edge time 3310, into FIFO input 3320. A software timer or othermechanism runs at a scheduled time determined by removing a time fromthe FIFO output 3330. When the scheduled time arrives, the state of theoutput signal is changed and a new scheduled time is removed from theFIFO.

Judges, Outputs, and Examples of Use

FIG. 34 shows parameter views for user configuration of the ObjectDetectJudge. A first parameter view 3490 is used for visual event detectionmode, and a second parameter view 3492 is used for external triggermode.

Presentation control 3400 allows selection of either indexed orcontinuous object presentation.

Frame filtering controls 3410 allow the user to set limits on the countor total object detection weight of active frames. Minimum frame spinner3412 allows a user to choose the minimum required count or weightthreshold, as explained above in the description of FIG. 12 where itrefers to FIG. 34. Maximum frame spinner 3414 allows a user to choose amaximum frame count or weight, after which dynamic image analysis willterminate and a decision will be made regardless of how many more activeframes may be found. This allows the user to limit the amount of timethe invention will spend on one object, and is particularly useful forindexed or slow-moving objects.

Idle time controls 3420 allow a user to specify minimum and maximumtimes for idle step 580 (FIG. 5). If the minimum and maximum are equal,the idle time is fixed by the user. If the maximum is greater than theminimum, the vision detector can automatically choose a time within thespecified range based on the measured rate at which objects are beingpresented. A phase-locked loop, shown in FIGS. 47 and 48, can be used tomeasure the object presentation rate. In an illustrative embodiment, theautomatically selected idle time is half the object presentation period,clamped if necessary to fall within the user-specified range.

Missing frame spinner 3430 allows a user to specify the maximum numberof consecutive inactive frames that will be accepted without terminatingdynamic image analysis. Such a frame is illustrated by analysis step 542in FIG. 5.

Marking control 3440 allows a user to select the marking mode. Ifmarking by location is selected, the user must specify a Locator usinglocator list control 3442.

Output mode control 3450 allows a user to select the mode that defineswhen a pulse appears on the logic output.

Frame count spinner 3460 allows a user to select the number of frames toanalyze in external trigger mode.

FIG. 35 shows a parameter view for user configuration of the ObjectPassJudge. Mode control 3500 allows a user to choose between the weightedaverage method described for FIG. 12, and the percentile methoddescribed for FIG. 13. If the percentile method is chosen, the thresholdt therein described can be selected using percentile spinner 3510.

FIG. 36 shows a parameter view for user configuration of an OutputGadget. Mode control 3600 allows a user to choose how the output signalis controlled. In “straight through” mode, the logic input is passeddirectly to the output signal without any delay or synchronization. In“delayed” mode, on the rising edge of the logic input an output pulse isscheduled to occur at a time delayed from the most recently recordedmark time (or encoder count) by the amount specified by delay controls3610, and of duration specified by pulse controls 3620. The scheduledpulse may be placed in a FIFO associated with the Output Gadget.

FIG. 37 illustrates one way to configure the present invention tooperate in visual event detection mode when connected to a PLC. A logicview shows ObjectDetect Judge 3720 wired to “DetectOut” Output Gadget3730 and ObjectPass Judge 3740 wired to “PassOut” Output Gadget 3750. Anappropriate set of Photos and Gates is wired to the Judges and notshown.

ObjectDetect Judge 3720 is configured for continuous presentation, markby location, and output when done, as shown in FIG. 34.

A first parameter view 3700 shows that “DetectOut” Output Gadget 3730 isconfigured to generate a synchronized 10 ms. pulse 25 ms. after the marktime. This pulse will be triggered by the rising edge of ObjectDetectJudge 3720. A second parameter view 3710 shows that “PassOut” OutputGadget 3750 is configured to send the pass/fail result from ObjectPassJudge 3740 directly to its output signal.

A PLC could sense the two output signals, noting the time of the risingedge of the signal from “DetectOut” and latching the output of “PassOut”on that edge. The PLC would then know that an object has been detected,when it crossed the mark point (25 ms. before the rising edge of“DetectOut”), and the result of the inspection.

FIG. 38 illustrates one way to configure the invention to operate invisual event detection mode for direct control of a reject actuator. Alogic view shows AND Gate 3810 wired to the ObjectDetect and ObjectPassJudges so as to generate a pulse when a failing object is detected. ANDGate 3810 is wired to “RejectOut” Output gadget 3820, thereby receivingthis pulse. Parameter view 3800 shows that “RejectOut” Output Gadget3820 is configured to generate a synchronized 100 ms. pulse 650 encodercounts after the mark time. This pulse will be triggered by thedetection of a failed object. The ObjectDetect Judge is configured forcontinuous presentation, mark by location, and output when done.

FIG. 39 illustrates one way to configure the invention to operate inexternal trigger mode when connected to a PLC. A logic view shows theObjectDetect Judge wired to “DetectOut” Output Gadget and the ObjectPassJudge wired to “PassOut” Output Gadget, as was shown in FIG. 37.However, instead of wiring a set of Photos and Gates to the logic inputof the ObjectDetect Judge, “TrigIn” Input Gadget 3920 is used. It is thefact that the logic input of ObjectDetect depends on no Photos and atleast one Input Gadget that tells the Judge to operate in externaltrigger mode. If nothing is wired to ObjectDetect, it will operate incontinuous analysis mode.

First parameter view 3900 and second parameter view 3910 show that bothOutput Gadgets pass their logic inputs straight through to their outputsignals. A PLC could sense the two output signals and latch the outputof “PassOut” on the rising edge of “DetectOut”. The PLC would then bealerted that an object has been detected and the result of theinspection, and it would have to obtain the mark time, if needed, fromthe external trigger.

Another way to configure the invention to operate in external triggermode when connected to a PLC would be to use the Output Gadgetconfiguration of FIG. 37 to produce a synchronized output pulse. In thismethod the PLC could obtain the mark time from the vision detector andnot waste a valuable input on a connection to the external trigger.

FIG. 40 illustrates one way to configure the invention to operate inexternal trigger mode for direct control of a reject actuator. Operationis identical to the configuration shown and described for FIG. 38,except that “TrigIn” Input Gadget 4000 is wired to ObjectDetect insteadof a set of Photos and Gates. Note that the mark time is simply the timeof the external trigger.

Continuous Analysis and Examples

FIG. 41 illustrates one way to configure the invention to operate incontinuous analysis mode for detection of flaws on a continuous web.Image view 4110 shows a portion of continuous web 4100 that is movingpast the vision detector.

Locator 4120 and Edge Detector 4122 are configured to inspect the web.If the web breaks, folds over, or becomes substantially frayed at eitheredge, then Locator 4120 and/or Edge Detector 4122 will produce a falseoutput (logic value<0.5). If the web moves up or down Locator 4120 willtrack the top edge and keep Edge Detector 4122 in the right relativeposition to detect the bottom edge. However, if the width of the webchanges substantially, Edge Detector 4142 will produce a false output.

In a logic view “Top” Locator 4140 represents Locator 4120, and “Bottom”Detector 4150 represents Edge Detector 4122. These are wired to AND Gate4160, whose logic output is inverted and wired to “DetectOut” Outputgadget 4170. As can be seen in parameter view 4130, the inverted outputof AND Gate 4160 is passed directly to an output signal.

The output signal will therefore be asserted whenever either Photo'slogic output is false. The signal will be updated at the high frame rateof the vision detector, providing a continuous indication of the statusof the web.

FIG. 42 illustrates one way to configure the invention for detection offlaws on a continuous web that uses the ObjectDetect Judge for signalfiltering. Configuration is similar to that of FIG. 41, with image view4110 and parameter view 4130 applying to this Figure as well. A logicview shows the same configuration as FIG. 41, except that ObjectDetectJudge 4260 is placed between AND Gate 4160 and “DetectOut” Output Gadget4170.

Parameter view 4200 shows how the ObjectDetect Judge is configured. Inthis application there are no discrete objects, but an “object” isdefined to a stretch of defective web. The output mode is set to “outputwhen processing”, so one output pulse is produced for each stretch ofdefective web, whose duration is the duration of the defect.

By setting the minimum frame count to 3, insignificant stretches ofdefective web are filtered out and not detected. By allowing up to 3missing frames, insignificant stretches of good web immersed in a longerdefective portion are also filtered out. Thus the output signal will besimilar to that of FIG. 41, but with insignificant glitches removed.Note that these values are intended to be exemplary, and would be chosenby a user to be suitable for a specific application.

There is no maximum frame count specified, so a stretch of defective webcan continue indefinitely and be considered one defect. The idle timemay be set to zero so that the web is always being inspected.

FIG. 43 illustrates one way to configure the invention for detection offlaws on a continuous web that uses the ObjectDetect Judge and an OutputGadget for signal synchronization. Configuration is identical to FIG.42, except that “DetectOut” Output Gadget 4170 is configured as shown inparameter view 4300.

Note in parameter view 4200 that the ObjectDetect Judge is configured toset the mark time to the time that the “object”, here a stretch ofdefective web, is first detected. As can be seen in parameter view 4300,a 50 ms. output pulse is generated 650 encoder counts after the marktime.

Detecting Objects Without Using Locators

FIG. 44 illustrates one way to configure the invention to operate invisual event detection mode without using Locators as input to theObjectDetect Judge to determine active frames. An object 4400 containinga feature 4410 moves left to right in the FOV of a vision detector.Contrast Detector ROI 4420 is positioned to detect the presence offeature 4410. A left Brightness Detector ROI 4430 and a right BrightnessDetector ROI 4432 are positioned to detect the object over a range inthe FOV defined by their separation.

A logic view shows “Right” Brightness Detector 4440 corresponding toright Brightness Detector ROI 4432, and “Left” Brightness Detector 4442corresponding to left Brightness Detector ROI 4430. “Right” BrightnessDetector 4440 produces a true logic output when object 4400 is notcovering right Brightness Detector ROI 4432, because the background inthis example is brighter than object 4400. “Left” Brightness Detector4442 produces a true logic output when object 4400 is covering leftBrightness Detector ROI 4430, because its output is inverted. ThereforeAND Gate 4460 produces a true logic output when the right edge of object4400 is between left Brightness Detector ROI 4430 and right BrightnessDetector ROI 4432.

Note that the logic output of AND Gate 4460 is actually a fuzzy logiclevel that will fall between 0 and 1 when the right edge of object 4400partially covers either ROI. Contrast Detector ROI 4420 must be largeenough to detect feature 4410 over the range of positions defined by theseparation between left Brightness Detector ROI 4430 and rightBrightness Detector ROI 4432, because since no locators are being usedit will not be moved.

AND Gate 4460 is wired to the ObjectDetect Judge, and “Hole” ContrastDetector 4470, corresponding to Contrast Detector ROI 4420, is wired tothe ObjectPass Judge. The Judges in this example are configured forvisual event detection and direct control of a reject actuator, as inFIG. 38.

Note that in the example of FIG. 44, a Locator could readily have beenused instead of the pair of Brightness Detectors, because the right edgeof object 4400 is relatively straightforward to locate. There will beapplications, however, where locators may be difficult to use due tolack of clean edges, and where using some combination of Detectors wouldbe advantageous.

Learning

FIG. 45 illustrates rules that may be used by an illustrative embodimentof a vision detector for learning appropriate parameter settings basedon examples shown by a user. This Figure will be used in conjunctionwith FIG. 22 to explain the exemplary learning method.

In an illustrative embodiment Photos can learn appropriate settings forthe sensitivity fuzzy thresholds. The learning process may also providesuggestions as to what Detectors to use and, where appropriate, whatsettings are likely to work best. Learning is by example—users willpresent objects that they have judged to be good or bad, and will soindicate by interaction with HMI 830.

Learning is optional, as default or manual settings can be used, but isstrongly recommended for the Brightness and Contrast Detectors becausetheir analog outputs are in physical units (e.g. gray levels) that haveno absolute meaning. Learning is less critical for the Edge, Spot, andTemplate Detectors, because their outputs are primarily based on anormalized correlation value that is dimensionless and with absolutemeaning.

For example, if an Edge or Spot Detector has an analog output of 80, onecan be fairly confident that an edge or spot has indeed been detected,because the correlation coefficient of the image ROI with an ideal edgeor spot template is at least 0.8. If the output is 25, one can be fairlyconfident that no edge or spot has been detected. But with, for example,a Brightness Detector, is 80 bright enough? Is 25 dark enough? This isbest learned by example in most instances.

There are two parts to the learning process. The first concerns howusers interact with HMI 830 to teach a particular example. The secondconcerns what a Photo does when presented with such an example.

Referring to FIG. 22, parameter views for Photos contain thumbs-upbutton 2210 and thumbs-down button 2212. The buttons are next to logicoutput label 2204, which, as previously described, may change color orother characteristic to distinguish between true (≧0.5) and false(<0.5). For purposes of the following discussion, green will be used fortrue and red for false. Although not visible in FIG. 22, thumbs-upbutton 2210 is green and thumbs-down button 2212 is red. The proximityto logic output label 2204 and matching colors helps the user understandthe function of the buttons.

Clicking green thumbs-up button 2210 means “learn that the logic outputof this Photo should now be green (true).” This operation is calledlearn thumbs-up. Clicking red thumbs-down button 2212 means “learn thatthe logic output of this Photo should now be red (false).” Thisoperation is called learn thumbs-down. These semantics are intended tobe clear and unambiguous, particularly when the ability to invert theoutput comes into play. The terms “good” and “bad”, often used indescribing example objects used for learning, change meaning dependingon whether the output is inverted and how it is used by Gates to whichit is connected.

Suppose one is using three Detectors that all must have a true outputfor an object to pass inspection. One can teach each Detectorindividually, but this could be unnecessarily cumbersome. If a goodobject is presented, all three Detectors should be true and so a singleclick somewhere to say, “this is a good object” would be useful. On theother hand, if a bad object is presented, there is often no way to knowwhich Detectors are supposed to be false, and so they are generallytaught individually.

Likewise, if the three Detectors are OR'd instead, meaning that anobject passes if any of them have a true output, then one might teach abad object with a single click, but for a good object the Detectors aregenerally trained individually. Once again, however, these rules changeas inputs and outputs are inverted.

Teaching multiple detectors with one click can be managed withoutconfusion by adding thumbs-up and thumbs-down buttons to Gates,following rules shown in FIG. 45. The buttons are displayed and clickedin a parameter view for each Gate, but FIG. 45 shows them in a logicview for descriptive purposes. Once again, clicking a green (red)thumbs-up (-down) button tells the Gate to learn thumbs-up (-down),which means “learn that the logic output of this Gate should now begreen (red).”

An AND Gate with a non-inverted output learns thumbs-up by tellingGadgets wired to its non-inverted inputs to learn thumbs-up, and Gadgetswired to its inverted inputs to learn thumbs-down. For example,non-inverted AND Gate 4500 learns thumbs-up by telling Photo 4502 tolearn thumbs-up, AND Gate 4504 to learn thumbs-up, and Photo 4506 tolearn thumbs-down. An AND Gate with a non-inverted output cannot learnthumbs-down, so that button can be disabled and it ignores requests todo so.

An AND Gate with an inverted output learns thumbs-down by tellingGadgets wired to its non-inverted inputs to learn thumbs-up, and Gadgetswired to its inverted inputs to learn thumbs-down. For example, invertedAND Gate 4510 learns thumbs-down by telling Photo 4512 to learnthumbs-up, Photo 4514 to learn thumbs-up, and OR Gate 4516 to learnthumbs-down. An AND Gate with an inverted output cannot learn thumbs-up,so that button can be disabled and it ignores requests to do so.

An OR Gate with a non-inverted output learns thumbs-down by tellingGadgets wired to its non-inverted inputs to learn thumbs-down, andGadgets wired to its inverted inputs to learn thumbs-up. For example,non-inverted OR Gate 4520 learns thumbs-down by telling OR Gate 4522 tolearn thumbs-down, Photo 4524 to learn thumbs-down, and Photo 4526 tolearn thumbs-up. An OR Gate with a non-inverted output cannot learnthumbs-up, so that button can be disabled and it ignores requests to doso.

An OR Gate with an inverted output learns thumbs-up by telling Gadgetswired to its non-inverted inputs to learn thumbs-down, and Gadgets wiredto its inverted inputs to learn thumbs-up. For example, inverted OR Gate4530 learns thumbs-up by telling Photo 4532 to learn thumbs-down, Photo4534 to learn thumbs-down, and AND Gate 4536 to learn thumbs-up. An ORGate with an inverted output cannot learn thumbs-down, so that buttoncan be disabled and it ignores requests to do so.

Photos that are told to learn thumbs-up or thumbs-down by a Gate act asif the equivalent button was clicked. Gates pass the learn command backto their inputs as just described. All other Gadgets ignore learningcommands in this example.

One exception to the above rules is that any Gate that has exactly oneinput wired to Photos, either directly or indirectly through otherGates, can learn thumbs-up and thumbs-down, so both buttons are enabled.Learn commands for such Gates are passed back to said input, inverted ifthe Gate's output or said input is inverted, but not both.

Users need not remember or understand these rules. In essence, the onlyrule to remember is to click the color one wants the output to be.Whenever the mouse is over a thumbs button, a tool tip could be providedto tell exactly which Photos will be trained, or explains why the buttonis disabled. When a thumbs button is clicked, a clear but non-intrusivefeedback can be provided to confirm that training has indeed occurred.

FIG. 46 illustrates how a Photo learns thumbs-up or thumbs-down. In anillustrative embodiment each Photo keeps two sets of statistics, calledthe output high statistics and the output low statistics. Each such setincludes a count, a sum of analog outputs, and a sum of squared analogoutputs, so that mean output low m_(lo), mean output high m_(hi),standard deviation of output low σ_(lo), and standard deviation ofoutput high σ_(hi) can be computed. For each learn command, a Photo willadd the analog output to a set of statistics as follows:

logic output logic output not inverted inverted thumbs-down output lowoutput high thumbs-up output high output low

In the example of FIG. 46, x-axis 4640 corresponds to the analog outputof a Photo. A distribution of output high values 4600, and adistribution of output low values 4610 have been learned fromappropriate example objects. Mean output low 4630, mean output high4620, standard deviation of output low (not shown), and standarddeviation of output high (not shown) have been computed.

The statistics are used to compute the Photo's sensitivity fuzzythreshold 4650, defined by low threshold to 1120 and high threshold t₁1122 (see also FIG. 11).t ₀ =m _(lo) +kσ _(lo)t ₁ =m _(hi) −kσ _(hi)  (21)

The parameter k may be chosen as appropriate. In an illustrativeembodiment, k=0.

The method for learning thumbs-up or thumbs-down is summarized by thefollowing steps:

-   -   1. Compute analog output and add to the output low or output        high statistics as appropriate for thumbs-up or thumbs-down, and        logic output inverted or not inverted;    -   2. Compute means and standard deviations for the output high and        output low statistics; and    -   3. Update the sensitivity fuzzy threshold based on the means and        standard deviations using equations 21.

Photos can also retain the most recent manual threshold settings, ifany, so that they can be restored if desired by the user. All statisticsand settings can be saved for all Photos on HMI 830, so that learningcan continue over multiple sessions. Users can clear and examine thestatistics using appropriate HMI commands.

If a set of statistics contains no examples, default values for the meancan be used. In an illustrative embodiment, the defaults arem _(lo) =a _(lo)+0.1(a _(hi) −a _(lo))m _(hi) =a _(lo)+0.9(a _(hi) −a _(lo))  (22)

where a_(lo) is the lowest possible analog output and a_(hi) is thehighest. If a set of statistics contains fewer than two examples, thestandard deviation can be assumed to be 0. This means that learning forma single example can be allowed, although it is generally notencouraged.

In another illustrative embodiment, a learn command for a Detectorcomputes the analog outputs that would result from each type of Detectoroperating on the ROI of the Detector being taught. Statistics for eachDetector type are computed and stored, and used to suggest betterDetector choices by looking for larger separations between the outputlow and output high examples.

Using a Phase-Locked Loop for Missing and Extra Object Detection

FIG. 47 illustrates the use of a phase-locked loop (PLL) to measure theobject presentation rate, and to detect missing and extra objects, forproduction lines that present objects at an approximately constant rate.The implementation of a PLL to synchronize to a train of pulses, and todetect missing and extra pulses, should be clear to those of ordinaryskill.

In one embodiment, a signal containing output pulses synchronized to themark time, for example output signal 3160 containing output pulse 3180(FIG. 31), is connected to the input of a conventional PLL. ConventionalPLL methods for missing and extra pulse detection are used to detectmissing and extra objects. The frequency of the voltage-controlledoscillator used by a conventional PLL gives the object presentationrate.

In an illustrative embodiment, a software PLL internal to visiondetector DSP 900 (FIG. 9) is used. Referring to FIG. 47, a sequence ofmark times 4700 is used by a software PLL to compute a correspondingsequence of time windows 4710 during which a mark is expected to occur.Time window 4720 contains no mark time, so a missing object is detectedat time 4730, corresponding to the end of time window 4720. A mark time4740 is detected outside of any time window, so an extra object isdetecting at time 4750 corresponding to mark time 4740. Informationabout object presentation rate, and missing and extra objects can bereported to automation equipment.

FIG. 48 illustrates the operation of a novel software PLL used in anillustrative embodiment. Shown is a plot of mark time on vertical axis4800 vs. object count on horizontal axis 4810. Each point plotted,including example point 4820, corresponds to one object detected. Toinsure numerical accuracy in the calculations, object count 0 can beused for the most recent object, and negative counts are used forprevious objects. The next expected object corresponds to count 1.

A best-fit line 4830 is computed using a weighted least-squares methodfurther described below. The weights are chosen to weight more recentpoints more strongly than more distant points. The slope of best-fitline 4830 gives the object presentation period, and the timecorresponding to count=1 gives the expected time of the next object.

In one embodiment, a fixed number of the most recent points are givenequal weight, and older points are given zero weight, so that onlyrecent points are used. The set of recent points being used is stored ina FIFO buffer, and well-known methods are used to update theleast-squares statistics as new points are added and old points areremoved.

In an illustrative embodiment, weights 4840 corresponding to the impulseresponse of a discrete Butterworth filter are used. A discreteButterworth filter is two-pole, critically-damped, infinite impulseresponse digital low-pass filter that is easily implemented in software.It has excellent low-pass and step response characteristics, considersthe entire history of mark times, has one adjustable parameter tocontrol frequency response, and does not need a FIFO buffer.

The output y_(i) at count i of a Butterworth filter with inputs x_(i) isv _(i) =v _(i−1) +f(x _(i) −y _(i−1))−f′v _(i−1)y _(i) =y _(i−1) +v _(i)  (23)where f is the filter parameter,f′=2√{square root over (f)}  (24)

and v_(i) are intermediate terms called velocity terms.

If the input is a unit impulse x_(o)=1, x_(i)=0 for i≠0, the output isthe impulse response, which will be referred to by the symbol w_(i). Theeffect of the filter is to convolve the input with the impulse response,which produces a weighted average of all previous inputs, where theweights are w_(−i). Weights 4840 are the values w_(−i) for f=0.12.

For the Butterworth PLL, three Butterworth filters are used,corresponding to the statistics needed to compute a least-squaresbest-fit line. Let the mark times be t_(i), where i=0 corresponds to themost recent object, i<0 to previous objects, and i=1 to the next object.Furthermore, let all times be relative to t₀, i.e. t₀=0. The followingweighted sums are needed:

$\begin{matrix}{{y_{t} = {\sum\limits_{i}{w_{- i}t_{i}}}}{y_{xt} = {\sum\limits_{i}{w_{- i}{it}_{i}}}}{y_{t\; 2} = {\sum\limits_{i}{w_{- i}t_{i}^{2}}}}} & (25)\end{matrix}$

Note that the summations are over the range −∞≦i≦0. These values areobtained as the outputs of the three Butterworth filters, which aregiven inputs t_(i), it_(i), and t _(i) ².

The following additional values are needed, and are derived from thefilter parameters f:

$\begin{matrix}{{y_{x} = {{\sum\limits_{i}{w_{- i}i}} = {1 - \frac{f^{\prime}}{f}}}}{y_{x\; 2} = {{\sum\limits_{i}{w_{- i}i^{2}}} = {y_{x} + \frac{6}{f}}}}{C = {\begin{pmatrix}y_{x\; 2} & y_{x} \\y_{x} & 1\end{pmatrix}^{- 1} = {\frac{f}{f^{\prime} + 2}\begin{pmatrix}1 & {- y_{x}} \\{- y_{x}} & y_{x\; 2}\end{pmatrix}}}}} & (26)\end{matrix}$

If best fit line 4830 is given by t_(i)=ai+b, then

$\begin{matrix}{\begin{pmatrix}a \\b\end{pmatrix} = {C\begin{pmatrix}y_{x\; t} \\y_{t}\end{pmatrix}}} & (27)\end{matrix}$

The value a is a very accurate measure of the current objectpresentation period. The equation of the line can be used to calculatethe expected mark time at any point in the near future. The weighted RMSerror of the best-fit line, as a fraction of object presentation period,is

$\begin{matrix}{E = \frac{\sqrt{y_{t\; 2} - \left( {{ay}_{xt} + {by}_{t}} \right)}}{a}} & (28)\end{matrix}$

which is a measure of the variability in the object presentation rate.

A new mark time to be input to the three filters occurs at time t₁,which is simply the elapsed time between the new object and the mostrecent at t₀=0. For each of the three Butterworth filters, adjustmentsmust be made to the output and velocity terms to reset i=0 and t₀=0 forthe new object. This is necessary for equations 26 to be correct, andalso for numerical accuracy. Without this correction, C would changewith every input. Here are equations for the three filters, showing thecorrections. Primes are used to indicate new values of the outputs andvelocity terms, and the u terms are temporary values.

$\begin{matrix}\begin{matrix}{v_{t}^{\prime} = {v_{t} + {f\left( {t_{1} - y_{t}} \right)} - {{f\;}^{\prime}v_{t}}}} \\{u_{t} = {y_{t} + v_{t}^{\prime}}} \\{y_{t}^{\prime} = {u_{t} - t_{1}}} \\{u_{x\; t} = {v_{xt} + {f\left( {t_{1} - y_{xt}} \right)} - {{f\;}^{\prime}v_{xt}}}} \\{v_{x\; t}^{\prime} = {u_{xt} - t_{1} - v_{t}^{\prime}}} \\{y_{xt}^{\prime} = {y_{xt} + u_{xt} - u_{t} - {y_{x}t_{1}}}} \\{u_{t\; 2} = {v_{t\; 2} + {f\left( {t_{1}^{2} - y_{t\; 2}} \right)} - {f^{\prime}v_{t\; 2}}}} \\{v_{t\; 2}^{\prime} = {u_{t\; 2} - {2\; t_{1}v_{t}^{\prime}}}} \\{y_{t\; 2}^{\prime} = {y_{t\; 2} + u_{t\; 2} - {2\; t_{1}u_{t}} + t_{1}^{2}}}\end{matrix} & (29)\end{matrix}$

A Butterworth PLL's output and velocity terms can be initialized tocorrespond to a set of values (a, b, E) as follows:

$\begin{matrix}\begin{matrix}{y_{t} = {{a\; y_{x}} + b}} \\{y_{xt} = {{a\; y_{x\; 2}} + {by}_{x}}} \\{y_{t\; 2} = {{ay}_{xt} + {by}_{t} + E^{2}}} \\{v_{t} = \frac{y_{t}}{y_{x}}} \\{v_{xt} = \frac{{f\left( {a + b - y_{xt}} \right)} - \left( {a + b + v_{t}} \right)}{f^{\prime}}} \\{v_{t\; 2} = \frac{{f\left\lbrack {\left( {a + b} \right)^{2} - y_{t\; 2}} \right\rbrack} - {2\left( {a + b} \right)v_{t}}}{f^{\prime}}}\end{matrix} & (30)\end{matrix}$

This allows one to reset the PLL to start at a particular period p usingequations 30 with a=p, b=0, and E=0. It also allows one to change thefilter constant while the PLL is running by initializing the output andvelocity terms using the current values of a, b, and E, and the newvalue of f.

In an illustrative embodiment, the Butterworth PLL determines whether ornot it is locked onto the input mark times by considering the number ofobjects that have been seen since the PLL was reset, and the currentvalue of E. The PLL is considered unlocked on reset and for the next nobjects. Once at least n objects have been seen, the PLL locks when Egoes below a lock threshold E₁, and unlocks when E goes above an unlockthreshold E_(u). In a further improvement, an unlocked filter parameterf_(u) is used when the PLL is unlocked, and a locked filter parameter f₁is used when the filter is locked. Switching filter parameters is doneusing equations 30 to initialize the output and velocity terms using thecurrent values of a, b, and E.

In an illustrative embodiment, n=8, E₁=0.1, E_(u)=0.2, f_(i)=0.05, andf_(u)=0.25.

If m consecutive objects are missed, the next object will arrive atapproximatelyt _(m+1) =a(m+1)+b  (31)

From this equation and the mark time t_(n) of a new object, one maydetermine m:

$\begin{matrix}{m = {{round}\left( {\frac{t_{n} - b}{a} - 1} \right)}} & (32)\end{matrix}$

To keep the equations for the Butterworth PLL correct, one must insertmark times for all of the missing objects. This may be accomplished byusing

$\begin{matrix}{t_{1} = \frac{t_{n}}{m + 1}} & (33)\end{matrix}$

m+1 times as input to equations 29.

It will be understood that any discrete low-pass filter can be used inplace of a Butterworth filter to implement a PLL according to thebest-fit line method of the present invention. Based on the methoddescribed above, equations 23 through 33 would be replaced asappropriate for the filter chosen.

The foregoing has been a detailed description of various embodiments ofthe invention. It is expressly contemplated that a wide range ofmodifications and additions can be made hereto without departing fromthe spirit and scope of this invention. For example, the processors andcomputing devices herein are exemplary and a variety of processors andcomputers, both standalone and distributed can be employed to performcomputations herein. Likewise, the imager and other vision componentsdescribed herein are exemplary and improved or differing components canbe employed within the teachings of this invention. Accordingly, thisdescription is meant to be taken only by way of example, and not tootherwise limit the scope of this invention.

I claim:
 1. A system for configuring an optoelectronic sensor having afield of view, comprising: a digital electronic memory, having aplurality of software elements stored therein, wherein each softwareelement performs an action; a plurality of connections, wherein eachconnection specifies that the action of a software element of theplurality of software elements depends on the action of at least oneother software element of the plurality of software elements; a firstsoftware element whose action analyzes a region of interest within thefield of view so as to produce a measure of evidence for a visiblecondition in the field of view, the first software element being one ofthe plurality of software elements; a second software element whoseaction makes a logic decision depending on the measure of evidence, thesecond software element being another of the plurality of softwareelements; a third software element whose action controls an outputsignal depending on the logic decision, the third software element beingyet another of the plurality of software elements; a first connectionthat specifies that the second software element depends on the firstsoftware element, the first connection being one of the plurality ofconnections; a second connection that specifies that the third softwareelement depends on the second software element, the second connectionbeing another of the plurality of connections; and a human-machineinterface comprising a display that shows using a ladder diagram of atleast one of the plurality of software elements and at least one of theplurality of connections, the ladder diagram having rungs, wherein partsof a rung have at least one open contact representing the measure ofevidence, and at least one icon representing the logic decision, whereinthe open contact representing the measure of evidence precedes the iconrepresenting the logic decision.
 2. The system of claim 1, wherein thehuman-machine interface further comprises means to manipulate and editthe ladder diagram so as to modify the plurality of connections.
 3. Thesystem of claim 1, wherein the display further shows using analternative representation at least one of the plurality of softwareelements and at least one of the plurality of connections; and thehuman-machine interface further comprises means to switch between theladder diagram and the alternative representation.
 4. The system ofclaim 3, wherein the alternative representation is a wiring diagram. 5.The system of any one of claims 1-4, wherein the optoelectronic visiondetector.
 6. The system of claim 1 wherein, at least one rung comprisesan open contact for the first software element that outputs the measureof evidence for a visible feature in the region of interest.
 7. Thesystem of claim 1, wherein the parts of the complete analysis maycomprise two or more of: inputs, measurements, logic decisions, andoutputs.
 8. Method for configuring an optoelectronic sensor andrepresenting the resulting optoelectronic program, the methodcomprising: providing a plurality of software elements stored on adigital memory; selecting, by a processor, a first software element tobe applied on a portion of an image for measuring evidence in theportion of the image, the first software element being one of theplurality of software elements; selecting, by the processor, a secondsoftware element that employs a logic decision for analyze the evidence,the second software element being one of the plurality of softwareelements; and generating instructions, by the processor, for displayinga ladder diagram on a HMI display, the ladder diagram having rungs, therungs having at least one open contact representing the evidence, and atleast one icon representing the logic decision, wherein the open contactrepresenting the measure of evidence precedes the icon representing thelogic decision, wherein parts of a rung represent portions of theoptoelectronic program.
 9. The method of claim 8, further comprising:selecting, by the processor, a third software element that producesoutput based on the logic decision, the third software element being oneof the plurality of software elements; and displaying on the ladderdiagram the third software element having as input one of the evidenceand the logic decision from at least one other of the plurality ofsoftware elements.
 10. The method of claim 9, wherein the optoelectronicsensor is a vision detector.
 11. The method of claim 8, furthercomprising providing, by the processor, a graphical tool on the HMIdisplay for manipulating and editing the ladder diagram so as to modifyone of the inputs and outputs to direct to a different one of theplurality of software elements.
 12. The method of claim 8, furthercomprising: providing, by the processor, an alternative representationat least one of the plurality of software elements and at least one ofthe plurality of connections; and providing a means based on the HMI toswitch between the ladder diagram and the alternative representation.13. The method of claim 12, wherein providing the alternativerepresentation comprises: providing, by the processor, a wiring diagramto represent the at least one of the plurality of software elements andthe at least one of the plurality of connections.
 14. The method ofclaim 8, wherein the optoelectronic sensor is a vision detector.
 15. Themethod of claim 8, wherein the parts of the optoelectronic program maycomprise two or more of: inputs, measurements, logic decisions, andoutputs.
 16. A computer program product for graphically representing anoptoelectronic program, the product tangibly embodied in anon-transitory computer readable medium, the computer program productcomprising instructions being operable to cause a data processingapparatus to: generate instructions for display of a ladder diagram ofthe optoelectronic program, the program comprising: calculating ameasure of evidence for a visible condition of a portion of a receivedimage, and computing a logic decision about a potential object in thereceived image based on the measure of evidence; and the ladder diagramcomprising one or more rungs, and having at least one open contactrepresenting the measure of evidence, and at least one icon representingthe logic decision about the potential object, wherein the open contactrepresenting the measure of evidence precedes the icon representing thelogic decision and wherein parts of a rung represent parts of theoptoelectronic program.
 17. The medium of claim 16, wherein the parts ofthe optoelectronic program may comprise two or more of: inputs,measurements, logic decisions, and outputs.
 18. The medium of claim 16,further comprising instructions for configuring and representing theoptoelectronic program, wherein the program is configured by a userusing an HMI interface.
 19. The medium of claim 16, further comprisinginstructions wherein the user can edit the ladder diagram withoutchanging the configuration of the program.
 20. The medium of claim 16,further comprising instructions wherein the user can hide any contactsof the optoelectronic program on the ladder.
 21. The medium of claim 16,further comprising instructions wherein computing a logic decision abouta potential object in the received image based on the measure ofevidence comprise computing presence of absence of the potential objectin the received image.